Wednesday, August 21, 2013

What is NFX? What is Unistack?

The name "NFX" stands for ".NET Framework Extension", there is also "JFX" and "*FX" in works. All of them form a "Unistack". "Unistack" = "Unified Software Stack", a conceptually-monolithic library that facilitates whatever service/facilities developers need to create scale-able, solid business/data-driven application systems (not just applications). As such, the unistack support many facilities that traditionally are scattered among myriads of disjoint software frameworks.

What is NFX?

NFX is a application-system development framework that addresses the following concerns in a UNIFIED (as in the "same pattern") way. Oh, and one more thing. NFX does not use any 3rd party libraries for the services it provides, except for things like database driver (i.e. MongoDB driver). So, NFX is a self-contained library (but for DB drivers) which is less than 2 mb in size compiled that provides:
  • Application Container - no counterpart in .NET framework
    • Provides a unified model (the same way of working with) for all application types: console, web, forms, service
    • Provides central hub for all application services: log, instrumentation, glue, throttling, security, zoned time etc.
    • Dependency injection container
    • Policy/Behavior injection container
    • Configuration management
    • Big Memory object database client - promotes stateful in-memory programming model for web and other server applications. Supports virtual out-of-process heaps of native CLR objects, supported by custom-purposed CLR-specific serialization mechanisms, with ability to transform object memory field structure between software upgrades
  • Code Analysis - no counterpart in .NET framework
    • Promotes run-time code/textual analysis that is very useful for pattern matching, serialization of dynamic formats and code inventorization
    • Provides general abstraction for languages, their lexers, parsers and semantic analyzers
    • Facilitates code analysis, provides pattern-matching language polymorphic Finite State Machine implementations
    • Organically supports non-copying serializers/deserializers of various text-based dynamic formats (XML, JSON, Erlang tuple stream, Laconic, etc.)
  • Data Abstraction Layer - replaces System.Data, ORMs (nHibernate, LINQ to SQL, Entity Framework)
    • Decouples business code from particular backing store implementations, supports SQL and NoSQL backends in the same way
    • Scaffolds RDB and NoSQL data, based on meta-information
    • Generates CRUD SQL automatically according to "Convention-over-Config" pattern, with ability to override any underlying provider statement
    • Abstracts non-homogeneous/hybrid data stores in a unified data-access service available within the application container
    • Support as of right now: PostgreSQL, MsSQLServer, MySQL, MongoDB. Planned: Oracle, DB2, Riak, CouchDB, Redis, Mnesia
    • DataStore providers may utilizes NATIVE CAPABILITIES of the target backend, developers can use backend-proprietary features without sacrificing performance (i.e. use CONNECT BY in ORACLE)
    • Ability to optionaly implement Query handlers in code i.e. when particular backend can not perform some action using built-in command
  • Environment / Configuration - compensates for .NET configuration framework deficiencies described here
    • Format-abstract configuration tree in memory
    • Non-file-based configurations (i.e. database configuration, cluster configuration, command-args-based configuration)
    • Supports XML and Laconic configuration file formats by default
    • Embed-able in Microsoft configuration system (i.e. web.config)
    • Full support for variable evaluation within configuration, path concatenation support (slashes)
    • Node navigation a la XPath (but works on any format including JSON and INI files)
    • Full support for imperative macro execution within configuration (IF, LOOP, SET)
    • Structural configuration merging, overrides, with rules (allow/stop/deny)
    • Aspect injection with Behaviors
    • Host environment variables support
    • APIs to configure classes (fields/properties) using attributes declaratively or imperatively
    • Plug-able macros for variable and function evaluation (i.e. DATE= :NOW)
    • Dependency injection class factory utils integrated
    • Unified way to configure any compoent in NFX (be it logger, CRUD data store, MVC, or anything else)
  • Inter-node/process Communication / Distributed Programming / Glue - replaces .NET WCF for cluster/grid programming
    • Promotes service-oriented architecture
    • Contract-based design
    • Synchronous and Asynchronous TCP/IP implementations (based on completion ports)
    • Extendable bindings - define protocol and channel-level interface
    • OneWay method calls, broadcasts
    • Blocking synchronous or asynchronous clients with CallReactor support
    • Message inspectors server and client-side
    • Contract / Server/Method level security support (authentication and authorization)
    • Allows for 100% stateful server programming with BigMemory support (in addition to typical stateless architecture)
    • Detailed instrumentation - bytes/transactions/calls/failures per node/binding/endpoint/contract/method
  • Instrumentation - much more flexible than .NET performance counters (i.e.write into cluster server or NoSQL db or file)
    • Events, gauges, levels, typed classes - Datum-based type safe classes checked at compile time
    • Multiple classifications of instruments (i.e. MyDatabaseFreePrimaryDisk = MyDatabaseIsntrument, IOInstrument, DataVolumeInstrument, etc. )
    • Asynchronous transparent acquisition of data - non performance affecting
    • Plug-able instrumentation backends (i.e. MongoDB, log file, Aum Cluster Server)
    • Application container log integration
    • Multidimensional histograms
  • Code Inventorization - no counterpart in .NET framework
    • Promotes decoration of types and methods with Inventory attribute
    • Allows to automatically discover various components of the system based on their logical classification (i.e. deployment risk)
    • Allows for creation of various automation tools that may generate/transform code based on some other code/metadata provided by inventorization
    • In cloud/cluster framework allows for automatic registration of components, services and their coupling
  • Various IO Formats - no such concept in .NET framework
    • Provides polymorphic StreamerFormat-derived classes that allow for efficient reading/writing of various binary terms
    • Supports unified parsing of binary format such as Slim, Erlang OTP etc.
    • Supports native Hadoop binary formatting (protocol)
  • Logging - no built-in library in .NET framework (replaces MS EntLib, Log4Net, NLog, etc.)
    • Asynchronous logging, priority logging
    • Sinks: MsSQLServer, SMTP, CSV file, debug file, MongoDB, Composite, AsyncComposite, *nix Syslog
    • Filters: Level ranges, days of week, hour ranges, date ranges, message patterns, custom filters (injectable filter statement)
    • SLA, Failures, Failovers (when A fails, log to B)
    • Flood prevention (log message time latch)
    • DEBUG/TRACE integration
    • Instrumentation integration
  • Record Model - MVVM not only for Web - no built-in library in .NET framework
    • Unified model for: Web, console, Forms, or anything else (Wpf, Silverlight etc.)
    • Allows developers to write complex business logic without considering the particular view/UI engine
    • Supports complex lookup scenarios - foreign keys selectable from a different model/screen
    • Various levels of validation: record-level, field-level, deferred
    • Field attributes: Enabled, Visible, Applicable, ReadOnly, Validated, Valid, Modified, Description, Watermark... and many more
    • Field validation attributes: Required, Default, Min/Max checking, Regexp, US Zip/Phone, Lookup Dictionaries, Char Casing, Passwords
    • Automatic DataStore CRUD (no need to write any SQL or NoSQL commands to load/save records) with ability to override default behaviors
    • Stateful programming - models may be retained between calls in BigMemory heap allowing for 100% stateful programming in volatile environments (i.e. on the Web)
    • Custom validation scripts for various targets (i.e. field/record-level validation JavaScript for browser)
  • Relational Model - no built-in library either .NET or any other major framework
    • Declarative well-structured programming language for database design (not only for RDBMS)
    • Database schema becomes parsable configuration file - i.e. can generate code, analyze dependencies etc. Can not parse ORACLE specific DDL now to generate C# code? How about MongoDB?(there is no DDL in MongoDB), however there is RelationSchema in NFX which is the same source be it ORACLE, MongoDB, Google BigTable or anything else that you may to want to model against
    • Pluggable backend compiler generates DDL/script for particular backends
    • Supports: tables, check constraints, keys, indexes, foreign keys, comments, verbatim scripts, domains, identities
    • Allows to execute macros - i.e.create many tables in a loop and make them inter-dependent
    • Outputs DDL statements of various kinds in either different or the single files
    • Supports data domains (even if backend does not support it) with pick-lists and range checks
    • Supports identities that result in different entities (Sequence in ORACLE, IDENTITY in MsSQL etc..)
    • Supports delta schema generation
  • Security - built-in support in .NET is very weak and hard to deal with (especially when there are 100s of permissions to check). .NEt permissions are called "Roles" and basically are boolean flags. In NFX permissions are complex vectors that may support integer levels of access along with any number of custom flags (i.e. "allow to print invoice, but hide The co-payor names")
    • Integrated with all components of NFX
    • Unified security model for: web, console, service, forms, wpf, etc.
    • Declaratively guard mathods/classes/actions, or imperatively check access levels by hand
    • Typed permissions, no need to type strings, type classes instead and get a compile-time error if mispelled
    • Imperative permissions, write addition security assertions in the permission - inversion of control principle
    • Designed to handle 1000s of permissions/grants per user - this is needed for systems with many screens where every field/button may need to be protected
    • Smart security descriptor caching /invalidation
    • Credentials/Identity/Tokens marshalling, transparent impersonation of users on remote hosts via Glued endpoints
    • Built for modern web/distributed systems: Integration with OAuth, Twitter, Facebook, LinkedIn and others using flexible Credentials class (that you can derive from)
  • Serialization - replaces various 3rd party libs working with JSON. Replaces DataContractSerializer for internal Glue programming
    • Slim Serialization - supports efficient binary serialization of CLR types. Implementation uses dynamic compilation of expression trees that yields much better (up to 15 times faster) performace than binary formatter. No need for data contracts.
    • JSON Serialization - very fast reading/writing objects without extra string copies. Perform JSON pattern matching over lexer-provided token stream (no need to parse the content if match does not happen). Output objects into JSON in a custom way (IJSONWritable).
    • Erlang native binary term serialization
    • Portable Object Documents - a format to express complex (including cyclical) native object graphs in normalized way (i.e. without reference cycles). Change object version and migrate objects using transforms
  • Service Model - no counterpart in .NET
    • Promotes a concept of a light-weight service - a "process" that may or may not own a thread/threads
    • Integrates in dependency injection application container. All major components in NFX are services
    • Provides message pumps/queues for asynchronous parallel programming
  • Templatization - in .NET only text templates are implemented for web an VS templates
    • Allows to create templates for any content generation (not only text, i.e. may create templatized images)
    • Provides text-based engine for generic document generation, and web documents. Performs 25% faster than classic ASP.NET pages and up to 40% faster than Razor
    • Web: may be served as pages or MVC views. Templates are 100% embed-able in DLL, no need to deploy files. No one will mend web page files by hand on a 100 server farm
  • Throttling - no counterpart in .NET
    • Declarative/imperative control of throughput
    • Time sliding/spacing throttles
    • Execution quotas (transactions per second, CPU% etc.)
  • Time Services - no counterpart in .NET as there are no "service" and "application" concepts
    • Facilitates distributed cluster-enabled programming in regions that span many time zones
    • Time Zoning in the application container. All components of the container may work in pre-configured zones (i.e. log time, transaction time, local time etc.). Zones defined by policies (possibly cluster-global)
    • Injectable external time sources (high precision/remote clocks)
    • Inter-component Time zone conversions
  • Web-specific MVC (Model View Controller) - supersedes ASP.MVC
    • Tiny implementation (less than 15 classes)
    • Routing support, controllers, actions, parameter binding, JSON support
    • 100% integrated with security/authorization/authentication - just tag action methods with permission attributes (that may be typed)
    • May use MVC without routing
    • Supports 100% stateful controllers (in addition to traditional stateless architecture) - controller instances may live in BigMemory heaps
    • Declarative control of controller lifecycle, timeouts, security, Http constraints etc.
    • Automatic payload parsing/assignment toform action methods
    • Integrated support for NFX.Templatization (return typed views etc.)
  • Web controls for RecordModel MVVM (auto building controls) - no similar concept in .NET or other major frameworks
    • Utilize metadata
    • Think about records and fields, not about divs and CSS
    • Override by hand in custom cases
    • Field views and grids
    • We support foreign keys (when they have to be looked-up from other screens with 100000 rows )
    • Declarative fields placement in HTML template markup
  • Windows forms controls for RecordModel - no similar concept in .NET or other major frameworks
    • Very similar to the controls on the Web
    • Utilize metadata
    • Thinks about records and fields, not boxes/buttons and style properties
    • Override by hand in custom cases
    • Field views, grids and forms
    • We support foreign keys (when they have to be looked-up from other screens with 100000 rows )

Wednesday, August 14, 2013

Aum Configuration as Facilitated By NFX

Problems with "built-in" approaches

Configuration is just another pain that everyone is constantly struggling with. I have never liked the .NET built-in configuration mechanisms. Why? Because, it does not have the so-needed functions that developers (we) have to compensate for daily. Although a typical .NET or Java developer is used to working with those limits, it is time to reconsider.

It is a fact of life - that many systems to this day (in 2013!!!!) use INI-like files with hand-written primitive string parsers. It is a total fiasco when it comes to supporting config files for DEV, TEST, SANDBOX environments.

Here, I have compiled a small list of things lacking in general config frameworks I have been working with in the past 15 years:

  • Centralized network configuration - what if I have to configure 10 servers? Do I copy files? How to keep all configs in one place, say SQL db?
  • Absurdly complex file locations, I once spent 1 hr trying to find the app.config file for some desktop app of mine on Windows Vista computer as installer put the file in the abyss or Profile..... folders
  • Microsoft .NET configuration framework has more than 100 classes, it is very complex yet very inflexible. Many people parse text files by hand
  • Inability to evaluate variables, i.e. in .NET there is no variable support in configuration, one must parse it by hand. INI files lack it. Registry is even more mess and hard to deal with
  • Painful APIs, i.e. an attempt to read a non-existing node must be always precluded by IF statements in .NET. Int values are hard to get as bool, dates as numbers etc...
  • Absence of unified configuration tree that could be hydrated from different sources, be it XML, INI, JSON, or even command-line args
  • And finally - the absence of unified configuration application - every component needs to configure itself "by hand"

Welcome to NFX Configuration

NFX library provides a unified configuration approach by providing:

  • Format-abstract configuration tree in-memory
  • Support for navigation (similar to x-path) within the tree
  • Variable evaluation, node inter-referencing, infinite cycles are detected and stopped
  • Environment variable (external vars) evaluation built-in
  • Pluggable variable value providers
  • Structural merging, overrides with rules. Prohibition (sealed sections) of overrides
  • Pluggable variable evaluation macros (i.e. ::NOW)
  • Support for XML, Laconic, Command-Line Args formats
  • Full support for imperative constructs (macros) - loops, vars, Ifs, blocks
  • Unified model to configure classes,services, properties,fields etc. from configuration
  • Aspect injection with configuration Behaviors - named kits of config values that may be applied to different nodes indirectly. This approach addresses cross-cutting concerns on the configuration level
  • Multiple getters for different nodes and data types (i.e. ValueAs: String/Date/Enum/Int.....) with defaults

Example of an XML-based configuration:






Example of a Laconic configuration with the same tree as the above:

       log-csv="NFX.Log.Destinations.CSVFileDestination, NFX"
       log-debug="NFX.Log.Destinations.DebugDestination, NFX"

         name="Logger" default-failover="destFailures"

        destination { type="$(/$log-csv)"
             filename="$(@/$log-root)$(::now fmt=yyyyMMdd)-$($name).csv.log"
             min-level="Info" }

        destination{  type="$(/$log-csv)"
             filename="$(@/$log-root)$(::now fmt=yyyyMMdd)-$($name).csv.log"
             max-level="PerformanceInstrumentation" }

        destination{ type="$(/$log-debug)"
             filename="$(@/$log-root)$(::now fmt=yyyyMMdd)-$($name).log"
             max-level="TraceZ" }

   }//nfx - notice the use of comments

Example of an command-line configuration used to inject some compiler settings. Yes, same framework for that:

    gluec "c:\mysrc\contacts" -assemblies NFX.dll MyBusiness.dll
                              -out "c:\templates\"
                              -opt single-file=true crlf=dos comments=false
                                   override=true base=MyPages.SimplePageBase
                              -sign name="Dmitriy" date=now 

Because all of the things above come down to the same tree in memory, we can use those completely disjoint ways of specifying settings in the same way to configure classes in code like so:

    /// Implements log destination that sends emails
    public class SMTPDestination : Destination
        public string SmtpHost { get; set; }
        public int SmtpPort { get; set; }

        public bool SmtpSSL { get; set; }

        public string FromAddress { get; set; }
        public string FromName { get; set; }

The configuration is a tree in memory, but how do we bind it to the actual code structures (properties, fields etc..)? How do we use it in our code? For that we have a number of ways.

  • Of course you can write regular code to bind any configuration value into any variable at runtime.
  • You can create Settings- derived type safe class that wraps configuration tree in a type-safe way. This is needed primarily for performance reasons when some tight code block may suffer from frequent access to text-based values that involve string parsing
  • You can use ConfigurationAttribute.Apply(object, section) method to apply config section data to some object
  • You can implement IConfigurable.Configure(section) in your class and apply section data by code to your class. This is useful for handling dynamic configuration structures when, for example a "parent" class manages many subordinate child classes that are polymorphic and may be injected by configuration

This topic organically touches inversion-of-control (really dependency injection) container of NFX.ApplicationModel. In Aum (and NFX) any process is hosted in a IApplication derivative, which injects all of its services into its own properties. There are many cases when one needs to configure their own particular components, for that FactoryUtils class is used which "manufactures" object instances as specified by the configuration and checks certain contract assertions supplied:

  /// Creates and configures an instance of appropriate configurable object
  /// as specified in supplied config node. Applies configured behaviors
  public static T MakeAndConfigure(IConfigSectionNode node, Type defaultType = null,
                                      IApplication application = null, object[] args = null)
            where T : IConfigurable

  protected override void DoConfigure(IConfigSectionNode node)

     foreach (var snode in 
                  node.Children.Where(n =>
         RegisterSink( FactoryUtils.MakeAndConfigure(snode) as LogSink );

Aum Cluster Configuration

Aum Clusterware is built on top of NFX (on CLR platform), so it is all based on the NFX.Environment.Configuration classes, however it takes that configuration capabilities to the next level - the hierarchical configuration tree that the cluster is configured from. Because of it, we do not need to maintain configuration for 1000s of servers, we project configuration segments down the tree to arrive to final configuration section that is built for particular node/service/end point. We do come all the way down to endpoint level.

This topic definitely deserves a post of its own that i'll do in the next month.