On my notebook for Visualization, it is perfect which the months are sorted correctly, as it is ordered by month_numeric. However, when I bring over to the Dashboard, its month is not sorted in the order which I need (see below). Where do I miss some...
Hi,After 48 hours after passing the exam, I didn´t receive any certification. My ID: E-03DK31. This is the prevous ticket Ticket URL: https://help.databricks.com/s/case/5008Y00002AAdImQAL
Please provide more information about Photon
There is some information here, in case you haven't checked yet, this may help gaining some knowledge. https://www.databricks.com/product/photon
Does anybody have any guidance on the best practices in setting up a DR env on a different region or on a different cloud ?
@ariforu cross region you can setup, but different cloud looks as of now it is not supported. did you get a chacne to go through this https://docs.databricks.com/administration-guide/disaster-recovery.html
I am trying to read from a databricks table. I have used the url from a cluster in the databricks. I am getting this error: java.sql.SQLDataException: [Simba][JDBC](10140) Error converting value to int.After these statements:jdbcConnUrl= "jdbc:spark:...
To create a Unity metastore the docs refer me to the account console in Databricks, see:https://docs.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/create-metastoreHowever when I go to manage account, I get redirected to select wo...
Please refer here - https://community.databricks.com/s/question/0D58Y000098lIqgSAE/unity-catalog-azure-account-console-how-to-accessYou must be an Azure Databricks account admin.The first Azure Databricks account admin must be an Azure Active Directo...
Hi, I'm a fairly new user and I am using Azure Databricks to process a ~50Gb JSON file containing real estate data. I uploaded the JSON file to Azure Data Lake Gen2 storage and read the JSON file into a dataframe.df = spark.read.option('multiline', '...
The spark connector is super slow. I found loading json into Azure cosmos dB then writing queries to get sections of data out was 25x times faster because cosmos dB indexes the json. You can stream read data from cosmosdb. You can find python code sn...
How refresh delta table with New raw from CDC Json file.
To refresh a delta table with new raw data from a CDC JSON file, you can use change data capture (CDC) to update tables based on changes in source data. Here are the steps:1. Create a streaming table using the CREATE OR REFRESH STREAMING TABLE statem...
Hello Everyone,I am trying to create Delta Live Table on top of csv file using below syntax:CREATE OR REFRESH LIVE TABLE employee_bronze_dltCOMMENT "The bronze employee dataset, ingested from /mnt/lakehouse/PoC/DLT/Source/."AS SELECT * FROM csv.`/mnt...
Hi @Manasi_Sarang ,I believe the Delta is unable to infer the schema as you are using select statement to read entire content from csv file and I think the inferschema won't work here. Instead you can try to create a temp live table or live view wit...
I have the following query;select customer_id, first(if(name_type = 'Official', name, null),true) official_name, first(if(name_type = 'Preferred', name, null),true) preferred_namefrom( select customer_id, ifnull(name_type, 'Official...
I experienced similar issues from time to time. What helped is to refresh the browser page.If that does not work, restart the sql warehouse.The internal error indeed is pretty vague, but my experience is that this is not related to a wrong SQL scrip...
So my team is using DBT for a lot of data lineage items but then at the data summit it was shown that data bricks also has a similar tool that is in fact better because it does lineage on columns. So what's the main draw of DBT at this point?
DBT (Data Build Tool) remains popular for its extensive templating capabilities, modularity, and open-source nature, which allows for customization and integration with various data platforms. While Databricks may offer more advanced lineage features...
Event covered everything a data engineer would dream of.My favorite discussions:- SparkConnect- AI on top unity catalog- delta live tables pipelines for streaming #Summit23
Amazing summit event, exciting features LLM, Lakeshouse IQ.
The deep learning training by Joseph and Marie was pretty cool
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
