- 2741 Views
- 3 replies
- 0 kudos
Foreign table to delta streaming table
I want to copy a table from a foreign catalog as my streaming table. This is the code I used but I am getting error: Table table_name does not support either micro-batch or continuous scan.; spark.readStream .table(table_name) ...
- 2741 Views
- 3 replies
- 0 kudos
- 0 kudos
What is the underlying type of the table you are trying to stream from? Structured Streaming does not currently support streaming reads via JDBC, so reading from MySQL, Postgres, etc are not supported. If you are trying to perform stream ingestion fr...
- 0 kudos
- 2082 Views
- 1 replies
- 0 kudos
cross join issue generating surrogate keys in delta table
I used below code to populate target table, it is working as expected but expect for surrogatekey column. After I inserted dummy entry -1 and did run merge code, it is generating the numbers in surrogatekey column like this 1,3,5,7,9 odd numbers, it ...
- 2082 Views
- 1 replies
- 0 kudos
- 0 kudos
This is expected. Identity columns can have gaps, or per the documentation: Values assigned by identity columns are unique and increment in the direction of the specified step, and in multiples of the specified step size, but are not guaranteed to b...
- 0 kudos
- 1491 Views
- 4 replies
- 2 kudos
Expose Databricks Cluster Logs to Grafana
I am trying to send Databricks cluster logs to Grafana using an init_script where I define the system journals to consume. The issue I am facing is that I cannot get the driver logs, standard error, and output to reach Grafana. Is there something spe...
- 1491 Views
- 4 replies
- 2 kudos
- 1012 Views
- 3 replies
- 0 kudos
Certification exam got suspended
Hello Team,My Data engineer associate exam got suspended within seconds of starting without any reason. After starting the exam screen got paused just after 10-20 seconds there was a notice that someone will contact you but no one contacted till the ...
- 1012 Views
- 3 replies
- 0 kudos
- 0 kudos
You need to raise a ticket with Databricks to get to a resolution: https://help.databricks.com/s/contact-us?ReqType=training%22%20%5Ct%20%22_blank
- 0 kudos
- 924 Views
- 1 replies
- 0 kudos
Liquid Clustering
How can I use column for liquid clustering that is not in first 32 column of my delta table schema.
- 924 Views
- 1 replies
- 0 kudos
- 0 kudos
We can only specify columns with statistics collected for clustering keys. By default, the first 32 columns in a Delta table have statistics collected. See Specify Delta statistics columns. We can use the below workaround for your use case: 1. Use th...
- 0 kudos
- 1247 Views
- 1 replies
- 0 kudos
Databricks migration
How easy to migrate from snowflake or redshift to Databricks ?
- 1247 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @RajPutta : it very easy to migrate any thing to Databrick only thing required is your team knowledge on databrick platform . below steps is important.DiscoveryAssessment Code conversion and Migration is very importPoCif you want to migrate fro...
- 0 kudos
- 4538 Views
- 2 replies
- 2 kudos
c# connector for databricks Delta Sharing
Any plans for adding c# connector? What are alternate ways in current state?
- 4538 Views
- 2 replies
- 2 kudos
- 2 kudos
I'm having problems getting the REST API calls for Delta Sharing to work. Python and Power BI work fine but the C# code that Databricks AI generates does not work. I keep getting an "ENDPOINT NOT FOUND" error even though config.share is fine.A C# con...
- 2 kudos
- 2356 Views
- 1 replies
- 0 kudos
Updates on a column in delta table with downstream autoloader
I've got the following questions:1. Can I pause autoloader jobs, delete cluster that was used to run these jobs, create a new cluster and run jobs with newer version cluster?2. I have one autoloader job that ingests JSONs and transforms this to a del...
- 2356 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello, 1.Yes you can pause the job, delete the cluster, upgrade versions of the cluster, etc. With Auto Loader and Structured Streaming the important thing is making sure that the checkpointLocation stays in tact, so no deletions, modifications, or m...
- 0 kudos
- 3045 Views
- 2 replies
- 0 kudos
scalar function in databricks
Hi Expert,here is sql server scalar function how to convert in databricks functionSQLCREATE function [dbo].[gettrans](@PickupCompany nvarchar(2),@SupplyCountry int, @TxnSource nvarchar(10),@locId nvarchar(50), @ExternalSiteId nvarchar(50))RETURNS INT...
- 3045 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Shree23 ,In Databricks, you can create scalar or tabular functions using SQL or Python. Here is the documentation .I converted your SQL Server function to Databricks standards. CREATE OR REPLACE FUNCTION gettrans( PickupCompany STRING, Sup...
- 0 kudos
- 2831 Views
- 2 replies
- 0 kudos
Enable system schemas
Hello All,I'm new with Databricks,Have an issue within enable system schemas. When run api call to check system schemas status in metastores -I see that all schemas in "Unavailable" state (except "information_schema", which is "ENABLE_COMPLETED").Is ...
- 2831 Views
- 2 replies
- 0 kudos
- 7807 Views
- 8 replies
- 2 kudos
Expose delta table data to Salesforce - odata?
HI Looking for suggestiongs to stream on demand data from databricks delta tables to salesforce.Is odata a good option?
- 7807 Views
- 8 replies
- 2 kudos
- 2 kudos
Hey, I think this might helphttps://www.salesforce.com/uk/news/press-releases/2024/04/25/zero-copy-partner-network/
- 2 kudos
- 6565 Views
- 7 replies
- 0 kudos
How to get databricks performance metrics programmatically?
How to retrieve all Databricks performance metrics on an hourly basis. Is there a recommended method or API available for retrieving performance metrics ?
- 6565 Views
- 7 replies
- 0 kudos
- 0 kudos
The spark logs are available through cluster logging. This is enabled at the cluster level for you to choose the destination for the logs. Just a heads up - interpreting them at scale is not trivial. I'd recommend having a read through the overwatch...
- 0 kudos
- 4071 Views
- 4 replies
- 1 kudos
an autoloader in file notification mode to get files from S3 on AWS -Error
I configured an autoloader in file notification mode to get files from S3 on AWS.spark.readStream\.format("cloudFiles")\.option("cloudFiles.format", "json")\.option("cloudFiles.inferColumnTypes", "true")\.option("cloudFiles.schemaLocation", "dbfs:/au...
- 4071 Views
- 4 replies
- 1 kudos
- 1 kudos
In case anyone else stumbles across this, I was able to fix my issue by setting up an instance profile with the file notification permissions and attaching the instance profile to the job cluster. It wasn't clear from the documentation that the file ...
- 1 kudos
- 7556 Views
- 4 replies
- 3 kudos
[DeltaTable] Usage with Unity Catalog (ParseException)
Hi,I'm migrating my workspaces to Unity Catalog and the application to use three-level notation. (catalog.database.table)See: Tutorial: Delta Lake | Databricks on AWSI'm having the following exception when trying to use DeltaTable.forName(string name...
- 7556 Views
- 4 replies
- 3 kudos
- 3 kudos
Thank you for the quick feedback @saipujari_spark Indeed, it's working great within a notebook with Databricks Runtime 13.2 which most likely has a custom behavior for unity catalog. It's not working in my scala application running in local with dire...
- 3 kudos
- 1982 Views
- 3 replies
- 0 kudos
Supporting Material for self phased Data Analysis with Databricks Course
Hi All,Newbie here, Any idea where I can find the supporting materials that are used in the online "Data Analysis with Databricks Course" that the instructor is using?. It seems to be having the scripts to create schema, tables, etcThanks in advance
- 1982 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @BSalla, here is our suggested learning path. I hope it helps you!
- 0 kudos
