Alex Rivera | Logout

What's a clean way to break up a DataTable into chunks of a fixed size with Linq?

Asked 2009-04-20T17:52:01.700
10

Update: Here's a similar question

Suppose I have a DataTable with a few thousand DataRows in it.

I'd like to break up the table into chunks of smaller rows for processing.

I thought C#3's improved ability to work with data might help.

This is the skeleton I have so far:

DataTable Table = GetTonsOfData();

// Chunks should be any IEnumerable<Chunk> type
var Chunks = ChunkifyTableIntoSmallerChunksSomehow; // ** help here! **

foreach(var Chunk in Chunks)
{
   // Chunk should be any IEnumerable<DataRow> type
   ProcessChunk(Chunk);
}

Any suggestions on what should replace ChunkifyTableIntoSmallerChunksSomehow?

I'm really interested in how someone would do this with access C#3 tools. If attempting to apply these tools is inappropriate, please explain!

Update 3 (revised chunking as I really want tables, not ienumerables; going with an extension method--thanks Jacob):

Final implementation:

Extension method to handle the chunking:

public static class HarenExtensions
{
    public static IEnumerable<DataTable> Chunkify(this DataTable table, int chunkSize)
    {
        for (int i = 0; i < table.Rows.Count; i += chunkSize)
        {
            DataTable Chunk = table.Clone();

            foreach (DataRow Row in table.Select().Skip(i).Take(chunkSize))
            {
                Chunk.ImportRow(Row);
            }

            yield return Chunk;
        }
    }
}

Example consumer of that extension method, with sample output from an ad hoc test:

class Program
{
    static void Main(string[] args)
    {
        DataTable Table = GetTonsOfData();

        foreach (DataTable Chunk in Table.Chunkify(100))
        {
            Console.WriteLine("{0}
c# linq c#-3.0 ienumerable
Edit
Report

1 Answer

0

Here's an approach that might work:

public static class Extensions
{
    public static IEnumerable<IEnumerable<T>> InPages<T>(this IEnumerable<T> enumOfT, int pageSize)
    {
        if (null == enumOfT) throw new ArgumentNullException("enumOfT");
        if (pageSize < 1) throw new ArgumentOutOfRangeException("pageSize");
        var enumerator = enumOfT.GetEnumerator();
        while (enumerator.MoveNext())
        {
            yield return InPagesInternal(enumerator, pageSize);
        }
    }
    private static IEnumerable<T> InPagesInternal<T>(IEnumerator<T> enumeratorOfT, int pageSize)
    {
        var count = 0;
        while (true)
        {
            yield return enumeratorOfT.Current;
            if (++count >= pageSize) yield break;
            if (false == enumeratorOfT.MoveNext()) yield break;
        }
    }
    public static string Join<T>(this IEnumerable<T> enumOfT, object separator)
    {
        var sb = new StringBuilder();
        if (enumOfT.Any())
        {
            sb.Append(enumOfT.First());
            foreach (var item in enumOfT.Skip(1))
            {
                sb.Append(separator).Append(item);
            }
        }
        return sb.ToString();
    }
}
[TestFixture]
public class Tests
{
    [Test]
    public void Test()
    {
        // Arrange
        var ints = new[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
        var expected = new[]
        {
            new[] { 1, 2, 3 },
            new[] { 4, 5, 6 },
            new[] { 7, 8, 9 },
            new[] { 10      },
        };

        // Act
        var pages = ints.InPages(3);

        // Assert
        var expectedString = (from x in expected select x.Join(",")).Join(" ; ");
        var pagesString = (from x in pages select x.Join(",")).Join(" ; ");

        Console.WriteLine("Expected : " + expectedString);
        Console.WriteLine("Pages    : " + pagesString);

        Assert.That(pagesStrin
answered 2009-09-26T17:31:05.683

Your Answer