- 2739 Views
- 2 replies
- 1 kudos
Azure Synapse vs Databricks
Hi team,Could you kindly provide your perspective on the cost and performance comparison between Azure Synapse and Databricks SQL Warehouse/serverless, as well as their respective use cases? Thank you.
- 2739 Views
- 2 replies
- 1 kudos
- 1 kudos
Agree with @mhiltner, it doesn't make sense to compare it with Synapse, as it's literally dead. You most likely want to compare it to Fabric instead. Fabric is highly under development, but IMHO it still lacks behind other Data/AI solutions. No catal...
- 1 kudos
- 41224 Views
- 3 replies
- 3 kudos
Resolved! How to import excel on databricks
To import an Excel file into Databricks, you can follow these general steps:1. **Upload the Excel File**:- Go to the Databricks workspace or cluster where you want to work.- Navigate to the location where you want to upload the Excel file.- Click on ...
- 41224 Views
- 3 replies
- 3 kudos
- 3 kudos
The question here is how to read the multi-excel files based on path.The mentioned solution interacts with one file only, do we have the ability to read all the Excel files in the folder?
- 3 kudos
- 1070 Views
- 3 replies
- 2 kudos
Resolved! Try Databricks sign up failed
Hi, I am trying to use Databricks with the community edition. However, when I tried to create an account, the sign-up failed after I completed the puzzle.
- 1070 Views
- 3 replies
- 2 kudos
- 5312 Views
- 6 replies
- 0 kudos
Streaming xls files Using Auto Loader
Hello,Is there a way to read .xls files using auto loader or is there any workaround since excel files are not supported by the auto loader per the following document?https://docs.databricks.com/en/ingestion/auto-loader/options.htmlThanks.
- 5312 Views
- 6 replies
- 0 kudos
- 0 kudos
I am facing the same issue--I have a stream that I'd like to use autoloader on with an .xlsx. Is there any update to any workarounds on this issue?
- 0 kudos
- 1358 Views
- 0 replies
- 0 kudos
File Not Found Error while reading pickle file
Hello, thereI have a pickle file uploaded in a mounted location in databricks ( /dbfs/mnt/blob/test.pkl). I am trying to read this pickle file using the below python snippetwith open(path + "test.pkl", "rb") as f: bands = pickle.load(f)But it t...
- 1358 Views
- 0 replies
- 0 kudos
- 5356 Views
- 2 replies
- 0 kudos
Resolved! Using private package, getting ERROR: No matching distribution found for myprivatepackage
My project's setup.py filefrom setuptools import find_packages, setup PACKAGE_REQUIREMENTS = ["pyyaml","confluent-kafka", "fastavro", "python-dotenv","boto3", "pyxlsb", "aiohttp", "myprivatepackage"] LOCAL_REQUIREMENTS = ["delta-spark", "scikit-lea...
- 5356 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, Does this look like a dependency error? All the dependencies are packed in the whl? Also, could you please confirm if all the limitations are satified? Refer: https://docs.databricks.com/en/compute/access-mode-limitations.html
- 0 kudos
- 2405 Views
- 2 replies
- 0 kudos
Resolved! Is DBFS going to be deprecated?
Is DBFS going to be deprecated? As I am using /dbfs/FileStore/tables/ location where a jar file is stored, and I am copying this jar file to /databricks/jars locations.My concerns is as DBFS root and mounts are deprecated, is that mean in coming days...
- 2405 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi Raphael,I am trying below init script to achieve this task, PFAAnd getting error as below,Cluster scoped init script abfss://container@storage.dfs.core.windows.net/init_script.sh failed: Failure to initialize configuration for storage account stor...
- 0 kudos
- 5880 Views
- 7 replies
- 0 kudos
Is it possible to view Databricks cluster metrics using REST API
I am looking for some help on getting databricks cluster metrics such as memory utilization, CPU utilization, memory swap utilization, free file system using REST API.I am trying it in postman using databricks token and with my Service Principal bear...
- 5880 Views
- 7 replies
- 0 kudos
- 0 kudos
At my company we are also interested in this feature, is there an ETA?
- 0 kudos
- 1823 Views
- 1 replies
- 2 kudos
How to update python's runtime on AWS lambda function
I heard that version 3.8 of Python on AWS Lambda will be EOL within the year. I would like to update this runtime, but where can I find the CloundFormation stack template.
- 1823 Views
- 1 replies
- 2 kudos
- 2 kudos
Thanks. I went to AWS Cloudformation stack and edited the template from python 3.8 to 3.12 and updated. I did this for both the workspace stack and the s3 ingestion stack. Will it break anything? Do I need to make any changes in the python code in th...
- 2 kudos
- 804 Views
- 1 replies
- 1 kudos
Error in notebook while execution
Error in callback <bound method UserNamespaceCommandHook.post_run_cell of <dbruntime.DatasetInfo.UserNamespaceCommandHook object at 0x7f5790c07070>> (for post_run_cell)
- 804 Views
- 1 replies
- 1 kudos
- 1 kudos
https://community.databricks.com/t5/data-engineering/error-in-notebook-execution/m-p/76226#M35165
- 1 kudos
- 7486 Views
- 2 replies
- 0 kudos
Callback bound method error
When executing a withColumn (running on DBR 14.3 LST) I get this error:Error in callback <bound method UserNamespaceCommandHook.post_run_cell of <dbruntime.DatasetInfo.UserNamespaceCommandHook object at 0x7feda2b2efb0>> (for post_run_cell):How shoul...
- 7486 Views
- 2 replies
- 0 kudos
- 0 kudos
We have the same issue using a shared cluster running DBR 14.3:Code executed: dfNew = dfTmp.withColumn(HashKeyColumnName, F.sha2(F.concat_ws("||", *ColumnList), 256))Error received: Error in callback <bound method UserNamespaceCommandHook.post_run_ce...
- 0 kudos
- 1535 Views
- 1 replies
- 0 kudos
When are DLT going to support multiple targets
Due to the limitations with all output data needing to be stored in one target we have stopped using DLT until more flexibility is added. If anyone has a workaround we are open to suggestions.
- 1535 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Zavi,One potential workaround is to establish multiple DLT pipelines, with each pipeline specifically configured to point to a unique target. This approach effectively allows for a diverse range of output data to be stored across various targets.T...
- 0 kudos
- 1369 Views
- 2 replies
- 1 kudos
Data in dataframe is also getting deleted when we are trying to delete records from underlying table
Hi , We are trying to load data from a delta table to a dataframe(a copy of original table) . Initially delta table has count 911 . The dataframe in which the data is loaded also has the same count .Now, we are deleting some records from the delta...
- 1369 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, There is a way to retain the copy of data frame, even if the data in underling table is manipulated but that's a memory expensive operation, be careful while using it.df1 = spark.createDataFrame(df.rdd.map(lambda x: x), schema=df.schema)Here we a...
- 1 kudos
- 1227 Views
- 1 replies
- 0 kudos
org.apache.spark.SparkException: Job aborted due to stage failure:
org.apache.spark.SparkException: Job aborted due to stage failure:
- 1227 Views
- 1 replies
- 0 kudos
- 0 kudos
Along with Job aborted due to stage failure: if you see slave lost... then it is due to less memory allocated for executors, more cores per executor more memory required or the other possibility is you have used max cpu available in cluster and the d...
- 0 kudos
- 1411 Views
- 1 replies
- 1 kudos
Resolved! In Azure Databricks, for when to use Kubernetes instead of Virtual Machines as compute backend?
- 1411 Views
- 1 replies
- 1 kudos
- 1 kudos
There is no distinction to make, it's VM's and you can't choose. Databricks SQL Serverless Warehouses uses K8s under the hood though.
- 1 kudos
