cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

YS1
by Contributor
  • 5126 Views
  • 6 replies
  • 0 kudos

Streaming xls files Using Auto Loader

Hello,Is there a way to read .xls files using auto loader or is there any workaround since excel files are not supported by the auto loader per the following document?https://docs.databricks.com/en/ingestion/auto-loader/options.htmlThanks.

Get Started Discussions
auto_loader
streaming
  • 5126 Views
  • 6 replies
  • 0 kudos
Latest Reply
Sicnarf
New Contributor II
  • 0 kudos

I am facing the same issue--I have a stream that I'd like to use autoloader on with an .xlsx. Is there any update to any workarounds on this issue?

  • 0 kudos
5 More Replies
divyasri1504
by New Contributor
  • 1119 Views
  • 0 replies
  • 0 kudos

File Not Found Error while reading pickle file

Hello, thereI have a pickle file uploaded in a mounted location in databricks ( /dbfs/mnt/blob/test.pkl). I am trying to read this pickle file using the below python snippetwith open(path + "test.pkl", "rb") as f:       bands = pickle.load(f)But it t...

  • 1119 Views
  • 0 replies
  • 0 kudos
vinitkhandelwal
by New Contributor III
  • 4940 Views
  • 2 replies
  • 0 kudos

Resolved! Using private package, getting ERROR: No matching distribution found for myprivatepackage

My project's setup.py filefrom setuptools import find_packages, setup PACKAGE_REQUIREMENTS = ["pyyaml","confluent-kafka", "fastavro", "python-dotenv","boto3", "pyxlsb", "aiohttp", "myprivatepackage"] LOCAL_REQUIREMENTS = ["delta-spark", "scikit-lea...

Get Started Discussions
dbx
package
private
python
  • 4940 Views
  • 2 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Does this look like a dependency error? All the dependencies are packed in the whl? Also, could you please confirm if all the limitations are satified? Refer:  https://docs.databricks.com/en/compute/access-mode-limitations.html 

  • 0 kudos
1 More Replies
ArvindDige
by New Contributor II
  • 1910 Views
  • 2 replies
  • 0 kudos

Resolved! Is DBFS going to be deprecated?

Is DBFS going to be deprecated? As I am using /dbfs/FileStore/tables/ location where a jar file is stored, and I am copying this jar file to /databricks/jars locations.My concerns is as DBFS root and mounts are deprecated, is that mean in coming days...

  • 1910 Views
  • 2 replies
  • 0 kudos
Latest Reply
ArvindDige
New Contributor II
  • 0 kudos

Hi Raphael,I am trying below init script to achieve this task, PFAAnd getting error as below,Cluster scoped init script abfss://container@storage.dfs.core.windows.net/init_script.sh failed: Failure to initialize configuration for storage account stor...

  • 0 kudos
1 More Replies
Prashanthkumar
by New Contributor III
  • 5555 Views
  • 7 replies
  • 0 kudos

Is it possible to view Databricks cluster metrics using REST API

I am looking for some help on getting databricks cluster metrics such as memory utilization, CPU utilization, memory swap utilization, free file system using REST API.I am trying it in postman using databricks token and with my Service Principal bear...

Prashanthkumar_0-1705104529507.png
  • 5555 Views
  • 7 replies
  • 0 kudos
Latest Reply
javierbg
New Contributor III
  • 0 kudos

At my company we are also interested in this feature, is there an ETA?

  • 0 kudos
6 More Replies
rt-slowth
by Contributor
  • 1660 Views
  • 1 replies
  • 2 kudos

How to update python's runtime on AWS lambda function

I heard that version 3.8 of Python on AWS Lambda will be EOL within the year. I would like to update this runtime, but where can I find the CloundFormation stack template.

  • 1660 Views
  • 1 replies
  • 2 kudos
Latest Reply
sandipkumar
New Contributor II
  • 2 kudos

Thanks. I went to AWS Cloudformation stack and edited the template from python 3.8 to 3.12 and updated. I did this for both the workspace stack and the s3 ingestion stack. Will it break anything? Do I need to make any changes in the python code in th...

  • 2 kudos
yeungcase
by New Contributor III
  • 633 Views
  • 1 replies
  • 0 kudos

Is it possible to configure a DLT pipeline continue to run when a single table ingestion failed?

Is it possible to configure a DLT pipeline continue to run when a single table ingestion failed?

  • 633 Views
  • 1 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Is it possible to configure a DLT pipeline continue to run when a single table ingestion failed?

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
kiranpeesa
by New Contributor
  • 675 Views
  • 1 replies
  • 1 kudos

Error in notebook while execution

Error in callback <bound method UserNamespaceCommandHook.post_run_cell of <dbruntime.DatasetInfo.UserNamespaceCommandHook object at 0x7f5790c07070>> (for post_run_cell)

  • 675 Views
  • 1 replies
  • 1 kudos
Latest Reply
Witold
Contributor III
  • 1 kudos

https://community.databricks.com/t5/data-engineering/error-in-notebook-execution/m-p/76226#M35165

  • 1 kudos
Henrik_
by New Contributor III
  • 6971 Views
  • 2 replies
  • 0 kudos

Callback bound method error

 When executing a withColumn (running on DBR 14.3 LST) I get this error:Error in callback <bound method UserNamespaceCommandHook.post_run_cell of <dbruntime.DatasetInfo.UserNamespaceCommandHook object at 0x7feda2b2efb0>> (for post_run_cell):How shoul...

  • 6971 Views
  • 2 replies
  • 0 kudos
Latest Reply
TjommeV-Vlaio
New Contributor III
  • 0 kudos

We have the same issue using a shared cluster running DBR 14.3:Code executed: dfNew = dfTmp.withColumn(HashKeyColumnName, F.sha2(F.concat_ws("||", *ColumnList), 256))Error received: Error in callback <bound method UserNamespaceCommandHook.post_run_ce...

  • 0 kudos
1 More Replies
Zavi
by New Contributor
  • 1478 Views
  • 1 replies
  • 0 kudos

When are DLT going to support multiple targets

Due to the limitations with all output data needing to be stored in one target we have stopped using DLT until more flexibility is added. If anyone has a workaround we are open to suggestions. 

  • 1478 Views
  • 1 replies
  • 0 kudos
Latest Reply
Rafael-Ribeiro
New Contributor II
  • 0 kudos

Hi Zavi,One potential workaround is to establish multiple DLT pipelines, with each pipeline specifically configured to point to a unique target. This approach effectively allows for a diverse range of output data to be stored across various targets.T...

  • 0 kudos
nikhilprajapati
by New Contributor
  • 1232 Views
  • 2 replies
  • 1 kudos

Data in dataframe is also getting deleted when we are trying to delete records from underlying table

  Hi , We are trying to load data from a delta table to a dataframe(a copy of original table) . Initially delta table has count 911 . The dataframe in which the data is loaded also has the same count .Now,  we are deleting some records from the delta...

nikhilprajapati_1-1701930598953.png nikhilprajapati_2-1701930598960.png nikhilprajapati_3-1701930598967.png nikhilprajapati_4-1701930598974.png
  • 1232 Views
  • 2 replies
  • 1 kudos
Latest Reply
Hkesharwani
Contributor II
  • 1 kudos

Hi, There is a way to retain the copy of data frame, even if the data in underling table is manipulated but that's a memory expensive operation, be careful while using it.df1 = spark.createDataFrame(df.rdd.map(lambda x: x), schema=df.schema)Here we a...

  • 1 kudos
1 More Replies
karola61
by New Contributor II
  • 1150 Views
  • 1 replies
  • 0 kudos

org.apache.spark.SparkException: Job aborted due to stage failure:

org.apache.spark.SparkException: Job aborted due to stage failure:

  • 1150 Views
  • 1 replies
  • 0 kudos
Latest Reply
rajeshg
New Contributor II
  • 0 kudos

Along with Job aborted due to stage failure: if you see slave lost... then it is due to less memory allocated for executors, more cores per executor more memory required or the other possibility is you have used max cpu available in cluster and the d...

  • 0 kudos
Vanshika
by New Contributor
  • 378 Views
  • 0 replies
  • 0 kudos

Databricks and Cloud Services Pricing

Hi,If I connect databricks (trial version) with AWS/Azure/Google Cloud and then work on dashboards and Genie - will there be any minimal charges, or its completely free to use the cloud services?

  • 378 Views
  • 0 replies
  • 0 kudos
FerArribas
by Contributor
  • 1213 Views
  • 1 replies
  • 1 kudos
  • 1213 Views
  • 1 replies
  • 1 kudos
Latest Reply
jacovangelder
Honored Contributor
  • 1 kudos

There is no distinction to make, it's VM's and you can't choose. Databricks SQL Serverless Warehouses uses K8s under the hood though. 

  • 1 kudos
fperry
by New Contributor II
  • 924 Views
  • 0 replies
  • 0 kudos

Concurrent State Update from Worker Nodes Possible?

For a data processing pipeline I use structured streaming and arbitrary stateful processing. I was wondering if the partitioning over several worker nodes and thus updating the state from different worker nodes has to be considered (e.g. using a lock...

  • 924 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels