AmitHegde.com

Rendering Chart in LinqPad

2016-01-23T00:00:00+00:00

Often charts tell us what numbers can’t tell us. Here is a quick and easy way to render charts in LinqPad:

void Main()
{
	new LpChart(series: GetSeries(), chartType: SeriesChartType.Column, width: 1250).Dump();
	new LpChart(series: GetSeries(), chartType: SeriesChartType.Line, width: 1250).Dump();
	new LpChart(series: GetSeries(), chartType: SeriesChartType.Bar, width: 500).Dump();
}

private ChartSeries GetSeries()
{
	var items = Enumerable.Range(1, 25).Select(x => string.Format("item-{0}", x));

	var rand = new Random(100);
	var prices = items.Select(x => rand.NextDouble());

	return new ChartSeries { XSeries = items.ToList(), YSeries = prices.ToList() };
}

public class ChartSeries
{
	public IList<string> XSeries { get; set; }
	public IList<double> YSeries { get; set; }
}

public class LpChart
{
	public Chart Chart { get; private set; }
	public ChartArea ChartArea { get; private set; }
	public Series Series { get; private set; }
	public Bitmap Bitmap { get; private set; }

	public LpChart(ChartSeries series, SeriesChartType chartType = SeriesChartType.Column, int width = 1200)
	{
		this.Chart = new Chart();
		this.ChartArea = new ChartArea();
		this.Chart.ChartAreas.Add(this.ChartArea);

		this.Series = new Series();
		this.Series.ChartType = chartType;
		this.Series.Points.DataBindXY(series.XSeries.ToArray(), series.YSeries.ToArray());
		this.Chart.Series.Add(this.Series);

		this.Chart.Width = width;

		this.Bitmap = new Bitmap(width: this.Chart.Width, height: this.Chart.Height);
	}

	public void Dump()
	{
		this.Chart.DrawToBitmap(this.Bitmap, this.Chart.Bounds);
		this.Bitmap.Dump();
	}
}

Here is a gist with all references.

LINQ Challenge #1

2015-07-30T00:00:00+00:00

In this challenge let us explore LINQ GroupBy based on multiple properties.

The problem is to Group By Property1 if it matches certain condition or Property2 and collect a meaningful key for each row. As earlier, I use LinqPad to get the code working quickly.

void Main()
{
	var list = new List<Person> {
		new Person { Parent1 = "SomeName1", Parent2 = "asdconst1fgh", Age = 10 },
		new Person { Parent1 = "SomeName2", Parent2 = "asdconst2fgh", Age = 10 },
		new Person { Parent1 = "SomeName1", Parent2 = "asdconst1fgh", Age = 10 } };

	var groupTypes = new List<string> { "Const1", "Const2" };

	//query syntax
	var gQuery1 = from data in list group data by GetKey(data, groupTypes);
	//method syntax
	var gQuery2 = list.GroupBy(data => GetKey(data, groupTypes));

	gQuery1.Dump();
	gQuery2.Dump();
}

//returns a meaningful key for each group
private string GetKey(Person data, List<string> groupTypes)
{
	//condition 1
	if (data.Parent1 == "SomeName1")
	{
		return data.Parent1;
	}
	
	//or condition 2
	return groupTypes.First(x => data.Parent2.ContainsOrdinalIgnoreCase(x));
}

//sample model
public class Person
{
	public string Parent1 { get; set; }
	public string Parent2 { get; set; }
	public int Age { get; set; }
}

public static class Extensions
{
	public static bool ContainsOrdinalIgnoreCase(this string str, string value)
	{
		return str.Contains(value, StringComparison.OrdinalIgnoreCase);
	}

	/// <summary>
	/// Performs a contains and takes a comparisonType (allows case insensitive comparison)
	/// </summary>
	public static bool Contains(this string str, string value, StringComparison comparisionType)
	{
		return !string.IsNullOrWhiteSpace(value) && str.IndexOf(value, comparisionType) >= 0;
	}
}

The crux of the problem lies in the method GetKey which returns a meaningful key for each row based on either Parent1 or Parent2.

Output from LinQPad:

Pretty Dump LINQPad Extension method for DynamicTableEntity

2015-07-10T00:00:00+00:00

Often, I write my Azure Storage Table queries on LINQPad and copy over to Visual Studio. I find this workflow faster as I can quickly execute the queries and see the results using the Dump() extension method. (Many people have tried to port it to Visual Studio and if you are looking for something similar look here and here)

I generally use DynamicTableEntity on LINQPad as it saves some time. But since DynamicTableEntity contains all the properties as IDictionary<string,EntityProperty> calling Dump() on it won’t print an easy to read output.

But wait, I discussed about ToDynamicList method I wrote for Azure Table Browser on this post. So passing the DynamicTableEntity collection through this method before dumping will convert it to a flat list. Quick and easy..!

But calling ToDynamicList() every time before calling Dump() is tedious. can we just override the original Dump() method? Well, I did not find a strait-forward way to do it; though, I kind of got it working.

public static void Dump(this object obj)
{
	if (obj.GetType() == typeof(List<DynamicTableEntity>))
	{
		var data = obj as List<DynamicTableEntity>;

		if (data == null)
		{
			obj.Dump();
		}
		else
		{
			data.ToDynamicList().Dump("Storage Table Entities");
		}
	}
	else
	{
		obj.Dump(obj.GetType().Name);
	}
}

One limitation here, is I need to do a ToList() before calling Dump() since the type is not IEnumerable<DynamicTableEntity> at run-time, easy hack was to do a ToList().

Paste this custom Dump() extension method on LINQPad’s My Extensions file along with ToDynamicList extension method, and resolve the dependencies. You are all set to use it. Here is a simple driver program to show how to use it:

void Main()
{
	var account = "";
	var key = "";
	var tableName = "";

	var cloudTable = GetStorageAccount(account, key).CreateCloudTableClient().GetTableReference(tableName);
	var tableQuery = new TableQuery<DynamicTableEntity>().Where(filter: string.Empty);

	var timer = Stopwatch.StartNew();
	
	//do a ToList() on it
	var data = cloudTable.ExecuteQuery(tableQuery).ToList();
	data.Dump();
	
	timer.Stop();
	timer.Dump();
}

static CloudStorageAccount GetStorageAccount(string accountName, string key, bool useHttps = true)
{
	var storageCredentials = new StorageCredentials(accountName, key);
	var storageAccount = new CloudStorageAccount(storageCredentials, useHttps: useHttps);
	return storageAccount;
}

I have created a gist with complete code here. And if you find a better way to override Dump(), do let me know in the comments… :)

Replacing Azure Storage Table Entity with Optimistic Concurrency

2015-07-06T00:00:00+00:00

Azure Storage Table offers Optimistic Concurrency Control (OCC) for TableEntities. Here is a detailed blog post on azure and concurrency control.

In this post let’s write a reusable C# method which replaces an entity using OCC, in the sense, if someone else has updated the entity before we could update it, we will have to pull the entity again, make our changes to the entity and write it to the storage.

In short, If the ETag maintained on the server is different from the one on the entity sent from client, azure storage service doesn’t update the entity, instead it throws an exception with HTTP status code 409.

Let us define the helper method to replace entity:

/// <summary>
/// Tries to replace an entity with Optimistic Concurrency Control with retry.
/// It updates the entity using the <see cref="entityUpdateAction"/> and tries to replace it on storage
/// If the replace fails (doesn't check for specific HTTP exception code), retrieves the entity again,
/// and tries to replace it on storage again after updating the properties using <see cref="entityUpdateAction"/>
/// </summary>
/// <typeparam name="T">Entity type</typeparam>
/// <param name="tableName">table name to replace entity</param>
/// <param name="entity">entity to replace</param>
/// <param name="entityUpdateAction"><see cref="Action"/> to update the entity properties</param>
/// <param name="retryCount">Retry count to try replacing. Default is 3</param>
public void ReplaceEntityOptimistically<T>(string tableName, T entity, Action<T> entityUpdateAction, int retryCount = 3)
    where T : TableEntitity, new()
	{
    for (int retryIndex = 0; retryIndex < retryCount; retryIndex++)
    {
        try
        {
            //update entity properties
            entityUpdateAction(entity);

            //try to replace
            ReplaceEntity(tableName, entity);

            break;
        }
        catch (Exception ex)
        {
            if (retryIndex == retryCount - 1)
            {
                Logger.LogException(ex,
                    "Could not Replace Entity. Failed Entity: {0}"
                        .ToFormat(entity));
                break;
            }

            //retrieve the entity again and try to replace it again
            entity = RetrieveEntity<T>(tableName, entity.PartitionKey, entity.RowKey);

            if (entity.IsNull())
            {
                break;
            }
        }
    }
}

public void ReplaceEntity<T>(string tableName, T entity) where T : TableEntity, new()
{
    CloudTable table = TableClient.GetTableReference(tableName);
    TableOperation tableOperation = TableOperation.Replace(entity);
    table.Execute(tableOperation);
}

As you can see, method ReplaceEntityOptimistically iterates over retryCount and tries to update the entity. If fails, it pulls a latest copy of the entity from server, makes necessary changes on the entity using entityUpdateAction and tries to update the entity again. RetrieveEntity method is a trivial method to retrieve the entity again from server.

Now, using this method is simple. All we need is an action to make changes to the entity properties. For example:

MyEntity myObject = GetEntity(); // get an entity from server

Action<MyEntity> entityUpdateAction = (entity) => { entity.SomeProp = "some value"; };
ReplaceEntityOptimistically("myTableName", myObject, entityUpdateAction);

Fixing Unmanaged DLL Not Found Exception On ASP.Net Application

2015-06-30T00:00:00+00:00

I often deal with DLL issues and I think it is good to document how I fix them as there would be many others facing similar issues. I assume that you have a fair knowledge of working of ASP.net application.. :)

I received a StackTrace today which said System.DllNotFoundException: Unable to load DLL '< dll name >'. Here is how I tracked it down and fixed it:

I looked through the application locally and nothing sounded suspicious, DLL in question was available on the bin directory.
This is an Azure WebRole running as a Azure Cloud Service. So I looked through the CSPKG by extracting its contents and the binary was included.
Since it is a Cloud Service, I could RDP in to the VM and the binary was available on the SiteRoot\0\bin directory.

So, the binary is available but IIS is unable to locate it. What could be wrong? Let’s figure it out.

The tool I use in this kind of situation is ProcMon from SysInternals. If ProcMon is new to you, go through this quick intro video by Scott Hanselman.

In continuation,

I copied over ProcMon to the VM on staging environment, which had only one VM running (if there are more VM’s you are not sure which VM wold get the reqeust from client…!).
I set ProcMon to capture file operations of w3wp.exe process and tried to send some requests to the URL which fails. Another level of filter is to make ProcMon show only rows that contain path with the binary in question.
I could see that IIS is probing for the binary in may directories, which all started with IIS temp directory which looks something like D:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\59a01799\5b6b1a2\assembly\dl3\a78168f6\7cabd0f6_d698d001\< binary name>. Other directories it probes are system directories and the ones mentioned on the PATH environment variable.
I looked at above the directory and the binary was not there. I copied over the file manually, to the above location and tried to send some requests to IIS.
I still got same exception in response…! What might still be wrong??? I checked through the ProcMon and now it was looking for some other DLL. Hmm.. this sounds interesting because IIS doesn’t say anything about this new binary.
I guessed that when IIS tries to load the first binary, there would be other dependencies for this binary and IIS is unable invoke certain methods which need such additional binaries and since it is an unmanaged binary, IIS just points to the original binary.
I used Dependency Walker to find out dependencies of the binary in question. There were three other binaries, which I copied to IIS temp from Site root. One important thing to remember while using Dependency Walker is, open the binary from the location where the binary exists, it will mark unavailable binaries in yellow color.
Now, the issue was fixed and the URLs started working fine.

Now, we just know the cause of the issue and know a rough way to fix it. But why would IIS copy all the binaries in to a temp location instead of executing it from site root? Short answer is IIS will copy resources to a directory (called Shadow Copying )which will be used by AppDomain and will have have a lock on the files to save itself from config file modifications and the like. This blog post has a nice and detailed info on Shadow Copying.

Since we know that the binary is available on site root, we can either add site root path to PATH environment variable (binary in question along with all its dependencies are available under SiteRoot directory) or we can just disable Shadow Copying by adding following web.coinfg entry:

<system.web>
	<hostingEnvironment shadowCopyBinAssemblies="false"/>
</system.web>

We need to remember that AppDomain will now consider SiteRoot as it’s location and modifications to files under SiteRoot would cause application to crash and not a good idea on production environments. But since I always deploy a full CSPKG to production environment and never touch the files on production environment, I am good with this change.

Well, that’s it we have fixed it successfully. I have tried to go as detailed as possible, though if you did not understand something, or have confusions, feel free to comment below, I will try to address them.

Writing a Fast and Easy to use Azure Storage Table Browser

2015-06-29T00:00:00+00:00

There are many tools that let you query data from Azure Storage Table including Visual Studio. But none of the tools are easy and fast as SQL Server Management Studio which most of us are comfortable with.

Though data looks very similar to the data on RDBMS once it’s on UI, none of the tools provide an intuitive user experience. Most of the tools I see around are either paid or if free, need lot of clicking around to get things done.

Half the world which uses Azure Storage uses Azure Storage Explorer, which doesn’t even let you copy text (authors of Azure Storage Explorer, if you are reading this don’t hesitate to fix it..!). Every developer I have come across is looking for a better tool, but most of them were asked to use this tool by a senior dev in the project and the lineage continued. Even I was handed over a copy of this tool and probably I used it for six months.

And one day I decided to write up my own tool out of the frustration of using this tool. For the readers who want to dive in to the code right away, here it is: Azure Table Browser. Executable Binaries here.

This is the first part of a tool which does CRUD operations on Azure Storage and supports read only operations on table as of now. If the tool gets some traction, I plan to open source more code… :)

In this post, I will explain about two important sections of this tool which make it useful and possible.

Converting table result received as DynamicTableEntity to a flat structure.

If you look through the class, all the properties of the entity (other than the default properties like PartitionKey) are exposed as

public IDictionary<string,EntityProperty> Properties { get; set; }

EntityProperty class exposes the value as set of properties, for example here is the code for BooleanValue property.

public bool? BooleanValue
{
get
{
	if (!this.IsNull)
	{
		this.EnforceType(EdmType.Boolean);
	}
	return (bool?)this.PropertyAsObject;
}
set
{
	if (value.HasValue)
	{
		this.EnforceType(EdmType.Boolean);
	}
	this.PropertyAsObject = value;
}
}

There are many other such properties such as DoubleValue, Int32Value etc. While pulling data through DynamicTableEntity we would not be knowing the types of each of the properties and this makes us go through a set of type comparisons before we can extract the value of the property.

I wrote a method ToDynamicList which does all these comparisons, extracts values for all the properties along with default properties (such as PartitionKey) and converts them to an ExpandoObject. Now this becomes a flat structure instead of being a two level structure. ToDynamicList returns resulting list as a List<dynamic> which can be bound to grid on the UI. The result could be passed through a Json serializer in case of a web application.

This method should have been made available on the Storage Library itself instead of making developers write this method, as dynamic type has been around for long time and libraries like dapper-dot-net have shown it to be pretty powerful. May be they would consider it if many request come in.. :)

Supporting client-side LINQ queries

Azure Storage supports OData query format which comes with a limited set of query operators. Though this works well for filtering data over HTTP, we can’t leverage on the advanced capabilities of LINQ to filter and aggregate the data. So I wrote a wrapper around this to perform LINQ operations on the data received from server. Here is the original project TextToLINQ I created, with some example usage. And the code if you want to look at the core of it right away.

The idea behind this class is simple and is reusable. given a compile-able c# LINQ expression

it will enclose it in a program
compiles an assembly
instantiates an object of the assembly
passes the collection to the assembly
collects the result (result can be projection of the current collection enabling aggregation)

Now on to Azure Table Browser..! Use it, Fork it and let me know how you feel.

Decoupling TableEntity while using Azure Storage

2015-06-28T00:00:00+00:00

Azure Storage has a nice documentation here and is easy to get started with.

All the examples on the documentation mention that Entities on Azure Storage map to classes derived from TableEntity. However, this would not fit well with some of the design goals. This post is all about how to abstract this requirement to derive each class from TableEntity and keep everything tidy.

TableEntity class implements ITableEntity and it has two methods to read/write data to the object. We will be leveraging the same to come up with a solution.

Let’s create a base class, which we can use to derive all Entity classes.

public class StorageTableEntityBase
{
	public string ETag { get; set; }

	public string PartitionKey { get; set; }
	public string RowKey { get; set; }
	public DateTimeOffset Timestamp { get; set; }
	
	public StorageTableEntityBase()
	{
	
	}
	
	public StorageTableEntityBase(string partitionKey, string rowKey)
	{
		PartitionKey = partitionKey;
		RowKey = rowKey;
	}
}

PartitionKey, RowKey, Timestamp and ETag are the default properties each entity on Azure Storage possesses. StorageTableEntityBase class defined above lets us to contain these properties, copied over from the entity.

Let us create an adapter class which implements ITableEntity:

internal class AzStorageEntityAdapter<T> : ITableEntity where T : StorageTableEntityBase, new()
{
	/// <summary>
	/// Gets or sets the entity's partition key
	/// </summary>
	public string PartitionKey
	{
		get { return InnerObject.PartitionKey; }
		set { InnerObject.PartitionKey = value; }
	}

	/// <summary>
	/// Gets or sets the entity's row key.
	/// </summary>
	public string RowKey
	{
		get { return InnerObject.RowKey; }
		set { InnerObject.RowKey = value; }
	}

	/// <summary>
	/// Gets or sets the entity's Timestamp.
	/// </summary>
	public DateTimeOffset Timestamp
	{
		get { return InnerObject.Timestamp; }
		set { InnerObject.Timestamp = value; }
	}

	/// <summary>
	/// Gets or sets the entity's current ETag.
	/// Set this value to '*' in order to blindly overwrite an entity as part of an update operation.
	/// </summary>
	public string ETag
	{
		get { return InnerObject.ETag; }
		set { InnerObject.ETag = value; }
	}

	/// <summary>
	/// Place holder for the original entity
	/// </summary>
	public T InnerObject { get; set; } 

	public AzStorageEntityAdapter()
	{
		// If you would like to work with objects that do not have a default Ctor you can use (T)Activator.CreateInstance(typeof(T));
		this.InnerObject = new T();
	}

	public AzStorageEntityAdapter(T innerObject)
	{
		this.InnerObject = innerObject;
	} 

	public virtual void ReadEntity(IDictionary<string, EntityProperty> properties, OperationContext operationContext)
	{
		TableEntity.ReadUserObject(this.InnerObject, properties, operationContext);
	}

	public virtual IDictionary<string, EntityProperty> WriteEntity(OperationContext operationContext)
	{
		return TableEntity.WriteUserObject(this.InnerObject, operationContext);
	} 
}

I have marked the class as internal to limit its access to current assembly. This class copies over properties such as PartitionKey, implements ReadEntity and WriteEntity methods to copy over rest of the properties to a temporary object InnerObject.

We can now define our Entity Class as follows:

	public class UserEntity : StorageTableEntityBase
	{
		public string UserName { get; set; }
		public string Email { get; set; }
	}

This is all the code we need to get it working. Now we can write a method to read from Storage Table as follows:

public T RetrieveEntity<T>(string tableName, string partitionKey, string rowKey)
        where T : StorageTableEntityBase, new()
{
    CloudTable table = TableClient.GetTableReference(tableName);
    TableResult tableResult = table.Execute(TableOperation.Retrieve<AzStorageEntityAdapter<T>>(partitionKey, rowKey));
    if (tableResult.Result != null)
    {
        return ((AzStorageEntityAdapter<T>)tableResult.Result).InnerObject;
    }
    return default(T);
}

I have not included the boilerplate methods instantiate TableClient, which is a trivial part and is available on the documentation.

Writing DynamicTableEntity to Azure Storage Table

2015-02-25T00:00:00+00:00

There are ample of samples available to show how to insert an object/entity to Azure Storage Table. However, all the samples inherit from TableEntity

This sample shows how to insert custom entities to table when we don’t have a class that inherits from TableEntity.

void Main()
{
	var account = "";
	var key = "";
	var tableName = "";

    var storageAccount = GetStorageAccount(account, key);
	var cloudTableClient = storageAccount.CreateCloudTableClient();
	var table = cloudTableClient.GetTableReference(tableName);
	
	var partitionKey = "pk";
	var rowKey = "rk";
	
	//create the entity
	var entity = new DynamicTableEntity(partitionKey, rowKey, "*", 
		new Dictionary<string,EntityProperty>{
				{"Prop1", new EntityProperty("stringVal")},
				{"Prop2", new EntityProperty(DateTimeOffset.UtcNow)},
			});
	
	//save the entity
	table.Execute(TableOperation.InsertOrReplace(entity));
	
	//retrieve the entity
	table.Execute(TableOperation.Retrieve(partitionKey,rowKey)).Result.Dump();
}

static CloudStorageAccount GetStorageAccount(string accountName, string key, bool useHttps = true)
{
	var storageCredentials = new StorageCredentials(accountName, key);
	var storageAccount = new CloudStorageAccount(storageCredentials, useHttps: useHttps);
	return storageAccount;
}

This code makes use of DynamicTableEntity which can take properties and values as IDictionary.

This code is written for LinQPad. Introduction on how to use Azure Storage Table can be found here.

Finding conflicting binary versions

2015-02-25T00:00:00+00:00

I deal with lot of .Net assemblies on a project I work with. Assemblies change their version and their dependencies frequently and one very common exception I see is:

Could not load file or assembly '...., Version=...., Culture=neutral, PublicKeyToken=....' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference.

We can handle this by using Assembly Redirection as long as there are no method signature changes. Adding assembly redirection can cause other side effects in some cases. If we want to change assemblies to reference same version of dependency, finding which assemblies refer this particular binary and which versions they refer becomes tricky.

Let’s write some code in C# to solve this:

void Main()
{
	var folder = @"";
	var files = Directory.GetFiles(folder,"*.dll",SearchOption.AllDirectories);
	
	var searchForNamespace = "";//Eg: Newtonsoft.Json
	
	if(files.Any())
	{
		foreach (var element in files)
		{
			try
			{	        
				var assembly = Assembly.LoadFile(element);
				if(assembly != null)
				{
					var references = assembly.GetReferencedAssemblies();
					if(references.Any(x => x.FullName.StartsWith(searchForNamespace)))
					{
						var referencedAssembly = references.Where(x => x.FullName.StartsWith(searchForNamespace));
						Console.WriteLine ("FileName: "+ element);
						Console.WriteLine ("Assembly: " + assembly.FullName + "\nReferenced Assembly: ");
						referencedAssembly.Dump();
					}
				}
			}
			catch (Exception ex)
			{
				//element.Dump("exception loading");
			}
		}
	}
}

This code is written for LinQPad. If you want to execute it on VS, LinqPad extension method .Dump() needs to be replaced.

Are private fields really private?

2015-01-02T00:00:00+00:00

As the saying goes, private fields and properties of a class are not available outside the class. But are they really private? Well, they are not; once they are in memory, anyone can tap in to them.

Let’s write some code in c#:

void Main()
{
	//create an instance and dump
	var demo = new PrivateDemo().Dump();
	
	FieldInfo[] privateFieldInfos = typeof(PrivateDemo)
				.GetFields(BindingFlags.NonPublic | BindingFlags.Instance);
	
	//set new value and dump
	foreach (var fieldInfo in privateFieldInfos)
	{
		fieldInfo.SetValue(demo, "I am no longer private...");
		fieldInfo.GetValue(demo).Dump(fieldInfo.Name);
	}
}

public class PrivateDemo
{
	public string PublicProperty { get; set; }
	private string PrivateProperty { get; set; }

	private string privateField;
	
	public PrivateDemo()
	{
		PublicProperty = "I am public...";
		PrivateProperty = "I am private...";
		privateField = "I am also Private...";
	}
}

And output from LinQPad: