- 1403 Views
- 0 replies
- 0 kudos
Is Unity Catalog enabled in AWS Government
Hey Databricks afficiandos, does anyone know if AWS Government cloud has Unity Catalog enabled?
- 1403 Views
- 0 replies
- 0 kudos
Forum Posts
- 2509 Views
- 0 replies
- 0 kudos
DLT table from a text source
I am trying to create a delta live table by reading a text source. I get an error message that states that both source and target should be in delta format. Am I missing something?
- 2509 Views
- 0 replies
- 0 kudos
- 1728 Views
- 0 replies
- 0 kudos
DBAcademy DLT cluster policy missing, &No permission to run Workspace-Setup
I am taking Data Engineering Associate course on the Databricks Partner Academy. To create pipeline for DLT, the cluster policy DBAcademy DLT is needed. I read from the previous community forums that "/Users/<YOUR USER NAME>/Data Engineering with Dat...
- 1728 Views
- 0 replies
- 0 kudos
- 7383 Views
- 5 replies
- 0 kudos
Incremental join transformation using Delta live tables
I'm attempting to build an incremental data processing pipeline using delta live tables. The aim to stream data from a source multiple times in a day and join the data within the specific increment only.I'm using autoloader to load the data increment...
- 7383 Views
- 5 replies
- 0 kudos
Latest Reply
- 0 kudos
basically you want to do a stream-stream join. If you want to do that you need to take a few things into account (see link).DLT might do this for you, but I never used it so I cannot confirm that.If your source tables are delta tables, you could ind...
- 0 kudos
- 9384 Views
- 4 replies
- 2 kudos
Hi All,I'm facing an issue with my Spark Streaming Job. It gets stuck in the "Stream Initializing" phase for more than 3 hours.Need your...
Hi All,I'm facing an issue with my Spark Streaming Job. It gets stuck in the "Stream Initializing" phase for more than 3 hours.Need your help here to understand what happens internally at the "Stream Initializing" phase of the Spark Streaming job tha...
- 9384 Views
- 4 replies
- 2 kudos
Latest Reply
- 2 kudos
I'm facing the same issue when I try to run this example Create a monitor using the API | Databricks on AWS (Inference Lakehouse Monitor regression example notebook). any idea?
- 2 kudos
- 2182 Views
- 1 replies
- 0 kudos
spline agent in Databricks use
spline Agent I use spline agent to get lineage of Databricks notebooks and for that i put the following code - attached to the notebook But i get the error attached%scalaimport scala.util.parsing.json.JSONimport za.co.absa.spline.harvester.SparkLinea...
- 2182 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Could be me but I do not see an error message?
- 0 kudos
- 5413 Views
- 4 replies
- 0 kudos
Resolved! INVALID_STATE: Storage configuration limit exceeded, only 11 storage configurations are allowed
Our team are trying to set up a new workspace (8th workspace), but failed to create the storage configurations required for the new workspace with an error of INVALID_STATE: Storage configuration limit exceeded, only 11 storage configurations are all...
- 5413 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
I solved the issue by simply going into Cloud Resources in Databricks console and navigated to "Credential Configuration" and "Storage Configuration" and deleted all the configurations which are not needed anymore(belongining to deleted workspaces)I ...
- 0 kudos
- 2446 Views
- 1 replies
- 0 kudos
Can not add comment on table via create table statement
I am not able to add comment using this create table statement with as (query)
- 2446 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
CREATE TABLE [ IF NOT EXISTS ] table_identifier [ ( col_name1 col_type1 [ COMMENT col_comment1 ], ... ) ] USING data_source [ OPTIONS ( key1=val1, key2=val2, ... ) ] [ PARTITIONED BY ( col_name1, col_name2, ... ) ] [ CLUSTERED B...
- 0 kudos
- 13065 Views
- 3 replies
- 3 kudos
Missing 'DBAcademy DLT' as a Cluster Policy when creating Delta Live Tables pipeline
I am currently in the middle of the Data Engineering Associate course on the Databricks Partner Academy. I am on module 4 - "Build Data Pipelines with Delta Live Tables", and trying to complete the lab "DE 4.1 - DLT UI Walkthrough". I have successful...
- 13065 Views
- 3 replies
- 3 kudos
Latest Reply
- 3 kudos
Policy will be available after running/Users/<YOUR USER NAME>/Data Engineering with Databricks - v3.1.4/Includes/Workspace-Setup
- 3 kudos
- 5557 Views
- 3 replies
- 0 kudos
Writing to Snowflake from Databricks - sqlalchemy replacement?
I am trying to migrate some complex python load processes into databricks. Our load processes currently use pandas and we're hoping to refactor into Spark soon. For now, I need to figure out how to alter our functions that get sqlalchemy connection e...
- 5557 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
@brian999 - spark-snowflake connector is inbuilt into the DBR. Please refer to the below article for examples. https://docs.databricks.com/en/connect/external-systems/snowflake.html#read-and-write-data-from-snowflake Please let us know if this hel...
- 0 kudos
- 3628 Views
- 1 replies
- 0 kudos
TaskSensor - check if task is succeded
Hi,I would like to check if the task within job is succeded (even the job is marked as failed because on of the tasks).I need to create dependency for tasks within other jobs. The case is that I have one job for loading all tables for one country. Re...
- 3628 Views
- 1 replies
- 0 kudos
- 3990 Views
- 3 replies
- 0 kudos
How to cache on 500 billion rows
Hello!I'm using a server less SQL cluster on Data bricks and I have a dataset on Delta Table that has 500 billion rows. I'm trying to filter to have around 7 billion and the cache that dataset to use it on other queries and make it run faster.When I ...
- 3990 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
I missed the 'serverless sql' part. CACHE is for spark, I don´t think it works for serverless sql.Here is how caching works on DBSQL.
- 0 kudos
- 5844 Views
- 4 replies
- 0 kudos
COPY INTO skipRows FORMAT_OPTIONS does not work
Based on the COPY INTO documentation, it seems I can use `skipRows` to skip the first `n` rows. I am trying to load a CSV file where I need to skip a few first rows in the file. I have tried various combinations, e.g. setting header parameter on or ...
- 5844 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
@yubin-apollo: My bad - I had the skipRows in the COPY_OPTIONS and not in the FORMAT_OPTIONS. It works, please ignore my previous comment. Thanks
- 0 kudos
- 25090 Views
- 3 replies
- 4 kudos
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 188.0 failed 4
When I am trying to read the data from sql server through jdbc connect , I get the below error while merging the data into databricks table . Can you please help whats the issue related to? : org.apache.spark.SparkException: Job aborted due to stage...
- 25090 Views
- 3 replies
- 4 kudos
Latest Reply
- 4 kudos
@rchauhan did you find a solution to the problem or know what settings caused the problem ?
- 4 kudos
- 4279 Views
- 4 replies
- 0 kudos
OPTIMIZE with liquid clustering makes filter slower than without OPTIMIZE
I created 15 Million records as a Delta Table and i'm running a simple filter query on that table based on one column value - which will return only one record. Because all the values on that column are unique.Delta Table is not partitioned.Before en...
- 4279 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
it seems that for this specific query Liquid Clustering has worse performance. It does not have better performance for all queries.The following are examples of scenarios that benefit from clustering:Tables often filtered by high cardinality columns...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 573 | |
| 349 | |
| 287 |
