- 2942 Views
- 1 replies
- 0 kudos
Updates on a column in delta table with downstream autoloader
I've got the following questions:1. Can I pause autoloader jobs, delete cluster that was used to run these jobs, create a new cluster and run jobs with newer version cluster?2. I have one autoloader job that ingests JSONs and transforms this to a del...
- 2942 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello, 1.Yes you can pause the job, delete the cluster, upgrade versions of the cluster, etc. With Auto Loader and Structured Streaming the important thing is making sure that the checkpointLocation stays in tact, so no deletions, modifications, or m...
- 0 kudos
- 4408 Views
- 2 replies
- 0 kudos
scalar function in databricks
Hi Expert,here is sql server scalar function how to convert in databricks functionSQLCREATE function [dbo].[gettrans](@PickupCompany nvarchar(2),@SupplyCountry int, @TxnSource nvarchar(10),@locId nvarchar(50), @ExternalSiteId nvarchar(50))RETURNS INT...
- 4408 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Shree23 ,In Databricks, you can create scalar or tabular functions using SQL or Python. Here is the documentation .I converted your SQL Server function to Databricks standards. CREATE OR REPLACE FUNCTION gettrans( PickupCompany STRING, Sup...
- 0 kudos
- 3515 Views
- 2 replies
- 0 kudos
Enable system schemas
Hello All,I'm new with Databricks,Have an issue within enable system schemas. When run api call to check system schemas status in metastores -I see that all schemas in "Unavailable" state (except "information_schema", which is "ENABLE_COMPLETED").Is ...
- 3515 Views
- 2 replies
- 0 kudos
- 9350 Views
- 8 replies
- 2 kudos
Expose delta table data to Salesforce - odata?
HI Looking for suggestiongs to stream on demand data from databricks delta tables to salesforce.Is odata a good option?
- 9350 Views
- 8 replies
- 2 kudos
- 2 kudos
Hey, I think this might helphttps://www.salesforce.com/uk/news/press-releases/2024/04/25/zero-copy-partner-network/
- 2 kudos
- 8139 Views
- 7 replies
- 0 kudos
How to get databricks performance metrics programmatically?
How to retrieve all Databricks performance metrics on an hourly basis. Is there a recommended method or API available for retrieving performance metrics ?
- 8139 Views
- 7 replies
- 0 kudos
- 0 kudos
The spark logs are available through cluster logging. This is enabled at the cluster level for you to choose the destination for the logs. Just a heads up - interpreting them at scale is not trivial. I'd recommend having a read through the overwatch...
- 0 kudos
- 5687 Views
- 4 replies
- 1 kudos
an autoloader in file notification mode to get files from S3 on AWS -Error
I configured an autoloader in file notification mode to get files from S3 on AWS.spark.readStream\.format("cloudFiles")\.option("cloudFiles.format", "json")\.option("cloudFiles.inferColumnTypes", "true")\.option("cloudFiles.schemaLocation", "dbfs:/au...
- 5687 Views
- 4 replies
- 1 kudos
- 1 kudos
In case anyone else stumbles across this, I was able to fix my issue by setting up an instance profile with the file notification permissions and attaching the instance profile to the job cluster. It wasn't clear from the documentation that the file ...
- 1 kudos
- 10213 Views
- 4 replies
- 3 kudos
[DeltaTable] Usage with Unity Catalog (ParseException)
Hi,I'm migrating my workspaces to Unity Catalog and the application to use three-level notation. (catalog.database.table)See: Tutorial: Delta Lake | Databricks on AWSI'm having the following exception when trying to use DeltaTable.forName(string name...
- 10213 Views
- 4 replies
- 3 kudos
- 3 kudos
Thank you for the quick feedback @saipujari_spark Indeed, it's working great within a notebook with Databricks Runtime 13.2 which most likely has a custom behavior for unity catalog. It's not working in my scala application running in local with dire...
- 3 kudos
- 2345 Views
- 3 replies
- 0 kudos
Supporting Material for self phased Data Analysis with Databricks Course
Hi All,Newbie here, Any idea where I can find the supporting materials that are used in the online "Data Analysis with Databricks Course" that the instructor is using?. It seems to be having the scripts to create schema, tables, etcThanks in advance
- 2345 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @BSalla, here is our suggested learning path. I hope it helps you!
- 0 kudos
- 1362 Views
- 2 replies
- 1 kudos
Finding Materials for Databricks Course
Hi All,Where can I find the supporting materials used by the instructor in the online "Data Analysis with Databricks" course? It appears to include scripts for creating schemas, tables, and other database structures.Thanks in advance.
- 1362 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @RoseCliver1 and @BSalla! To access the course supporting materials, please raise a support ticket here.
- 1 kudos
- 2100 Views
- 1 replies
- 1 kudos
Resolved! read file from local machine(my computer) and create a dataframe
I want to create a notebook and add a widget which will allow the user to select a file from local machine(my computer) and read the contents of the file and create a dataframe. is it possible? and how? in dbutils.widgets i dont have any options for ...
- 2100 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @khaansaab ,Currently, there is now out of the box mechanism that will allow you to that. As a workaround,you can create a UC volume and tell your users to upload files into that volume. Then you can create notebook that will have file_name parame...
- 1 kudos
- 1551 Views
- 1 replies
- 0 kudos
Merging customer and company account into single account
I have two accounts: One is my company account and another one is my personal account in databricks community. I want to merge it into single one. Kindly let me know how to do it.
- 1551 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @mahfooz_iiitian! Please send an email to community@databricks.com with both of your email addresses, specifying which account you’d like to retain. The IT team will assist you with merging the accounts.
- 0 kudos
- 1934 Views
- 2 replies
- 0 kudos
Is anyone using Databricks for Advanced HR Analytcs?
We are starting out with Databricks and I would like to use the tools to build out Advanced Analytics for HR.
- 1934 Views
- 2 replies
- 0 kudos
- 1120 Views
- 1 replies
- 0 kudos
Databricks Data engineer Exam got suspended while still 8 minutes left.
Hi @Cert-Team,I hope this message finds you well.Request ID- #00556592 I am writing to seek clarification regarding my recent exam, which was suspended due to a reflection issue caused by my spectacles. During the exam, the proctor paused it and aske...
- 1120 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Dharshan777 , We are sorry to hear that your exam was suspended.. Thank you for filing a ticket with our support team. Please allow the support team 24-48 hours for a resolution. In the meantime, you can review the following documentation: Beh...
- 0 kudos
- 17395 Views
- 3 replies
- 0 kudos
Space in Column names when writing to Hive
All,I have the following code.df_Warehouse_Utilization = ( spark.table("hive_metastore.dev_ork.bin_item_detail") .join(df_DIM_Bins,col('bin_tag')==df_DIM_Bins.BinKey,'right') .groupby(col('BinKey')) .agg(count_distinct(when(col('serial_lo...
- 17395 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi,I have faced this issue a few times. When we are overwriting the dataframes to hive catalog in databricks, it doesn't naturally allow for column names to have spaces or special characters. However, you can add an option statement to bypass that ru...
- 0 kudos
- 23528 Views
- 11 replies
- 1 kudos
Cannot create an account to try Community Edition
Hi,Whenever I try to signup for an account, I keep getting the following message - "an error has occurred. please try again later" when I click on the button "get started with databricks community edition".Could you please let me know why this could...
- 23528 Views
- 11 replies
- 1 kudos
- 1 kudos
I got the same problem if I try to register or login through Community Edition link. But I tried by clicking the "Try Databricks" button on top right corner of the https://www.databricks.com/ home page, I was able to register and login successfully j...
- 1 kudos
