- 1942 Views
- 2 replies
- 0 kudos
which cluster/worker/driver type is best for analytics work?
which cluster/worker/driver type is best for analytics work?
- 1942 Views
- 2 replies
- 0 kudos
- 0 kudos
Analytics work as in querying and analyzing data? Preferably using Databricks SQL? If so, then a SQL Warehouse is your best friend.
- 0 kudos
- 3534 Views
- 5 replies
- 2 kudos
Resolved! Delta sharing open protocol in Unity catalog: FileNotFoundError
Hi TeamI have created a recipient under delta sharing (azure databricks) . Unity catalog is enabled and data is stored in ADLS gen2. I have downloaded the credential file and trying to resue in my python script (as per databricks documentation) for a...
- 3534 Views
- 5 replies
- 2 kudos
- 2 kudos
I wasn't able to reproduce your issue. Is your delta table operable? can you see sample data from within databricks and query the table from within databricks? It almost looks like some parquet files are missing, causing your delta not queryable anym...
- 2 kudos
- 1167 Views
- 1 replies
- 0 kudos
Issues with Common Data Model as Source - different column size for blobs
I have a Dataverse Synapse link set up to extract data into ADLS gen2. I am trying to connect ADLS gen2 as the data source to read the data files in Databricks. I have CDC enabled for CDM Data with the partition of Year and Month.So, for example, if ...
- 1167 Views
- 1 replies
- 0 kudos
- 0 kudos
This is a thoughtful consideration, but have you considered using .option("mergeSchema", "true") when writing?Do keep in mind that this will affect the target table and possible downstream consumers. Ideally you want to have strict a schema contract ...
- 0 kudos
- 3644 Views
- 2 replies
- 0 kudos
Resolved! How can we connect to 2 different hive spark.hadoop.hive.metastore.uris
We need to read a table from 2 different spark.hadoop.hive.metastore.uris and do some validations.We are not able to connect to both spark.hadoop.hive.metastore.uris at the same time using sparkSession.I will be using Spark version: 3.1.1 and the lan...
- 3644 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi there @maskepravin02,We have once implemented this approach of two reading two different hive metasores, but it was not on AWS and GCP, maybe the docs can help.Though it is not recommended The best approach is to create separate spark applications...
- 0 kudos
- 1037 Views
- 1 replies
- 0 kudos
how do i pass the value from one task in the workflow to another task?
how do i pass the value from one task in the workflow to another task? #worksflow #orchestration
- 1037 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @pt07 , This may be what you're looking for. Can you please take a look? https://www.databricks.com/blog/2022/08/02/sharing-context-between-tasks-in-databricks-workflows.html
- 0 kudos
- 1326 Views
- 1 replies
- 0 kudos
- 1326 Views
- 1 replies
- 0 kudos
- 0 kudos
Delta Uniform is the way to go at the moment: https://www.databricks.com/blog/delta-lake-universal-format-uniform-iceberg-compatibility-now-ga In one word, tables are written in Delta but have compatibility with Iceberg as they will also save iceberg...
- 0 kudos
- 4458 Views
- 3 replies
- 2 kudos
- 4458 Views
- 3 replies
- 2 kudos
- 2 kudos
You could have a workflow with two tasks, one being a "trigger checker" that could be a super light task scheduled to run every X hours/minutes. This first task would check for your different triggers and define a success criteria for your next task....
- 2 kudos
- 480 Views
- 0 replies
- 0 kudos
Interesting talks at DAIS 2024
I feel some of the talks are quite interesting and inspiring. Great to see everyone here!
- 480 Views
- 0 replies
- 0 kudos
- 1015 Views
- 2 replies
- 1 kudos
Liquid Clustering
Love the new concept, will save a lot of compute and make queries a lot faster!
- 1015 Views
- 2 replies
- 1 kudos
- 366 Views
- 0 replies
- 0 kudos
Training sessions are helpful
Streaming and Unity catalog sessions are awesome
- 366 Views
- 0 replies
- 0 kudos
- 328 Views
- 0 replies
- 0 kudos
Data and AI Summit
The Data and AI Summit is San Francisco is wonderful!
- 328 Views
- 0 replies
- 0 kudos
- 954 Views
- 1 replies
- 0 kudos
DataAIsummit 2024
Having a wonderful learning experience and sharing experience and knowledge with fellow data engineers/scientists/architects
- 954 Views
- 1 replies
- 0 kudos
- 0 kudos
Interesting use cases so far. Looking forward ot the sessions tomorrow!
- 0 kudos
- 794 Views
- 0 replies
- 0 kudos
dbr
overwhelming and need to learn a lot
- 794 Views
- 0 replies
- 0 kudos
- 3490 Views
- 3 replies
- 1 kudos
SQL compilation error while connecting to snowflake from Databricks
Hi All,I'm trying to connect to the snowflake database from databricks notebook either to read the data or write the data. However I'm getting an weird error. The code and error are provided as follows, snowflake_table = (spark.read .format("snowfla...
- 3490 Views
- 3 replies
- 1 kudos
- 760 Views
- 0 replies
- 0 kudos
Partition Pruning with Non-Delta Files
Howdy - first time caller here.I'm trying to figure out how/if partition pruning works with non-CSV files. I have files landing in bronze in physical partitions of the form dlk_load_dtm=<load date time>. I'd like to load the partitions that don't yet...
- 760 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 486 | |
| 349 | |
| 287 |
