cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

divyasri1504
by New Contributor
  • 188 Views
  • 1 replies
  • 0 kudos

File Not Found Error while reading pickle file

Hello, thereI have a pickle file uploaded in a mounted location in databricks ( /dbfs/mnt/blob/test.pkl). I am trying to read this pickle file using the below python snippetwith open(path + "test.pkl", "rb") as f:       bands = pickle.load(f)But it t...

  • 188 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @divyasri1504 , Make sure you’re using the correct path to access the file. In Databricks, you should typically prefix everything with /dbfs (or dbfs:/ for native functions). Try using the full path like this: with open("/dbfs/mnt/blob/test.pkl...

  • 0 kudos
vinitkhandelwal
by New Contributor III
  • 2496 Views
  • 2 replies
  • 0 kudos

Resolved! Using private package, getting ERROR: No matching distribution found for myprivatepackage

My project's setup.py filefrom setuptools import find_packages, setup PACKAGE_REQUIREMENTS = ["pyyaml","confluent-kafka", "fastavro", "python-dotenv","boto3", "pyxlsb", "aiohttp", "myprivatepackage"] LOCAL_REQUIREMENTS = ["delta-spark", "scikit-lea...

Get Started Discussions
dbx
package
private
python
  • 2496 Views
  • 2 replies
  • 0 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, Does this look like a dependency error? All the dependencies are packed in the whl? Also, could you please confirm if all the limitations are satified? Refer:  https://docs.databricks.com/en/compute/access-mode-limitations.html 

  • 0 kudos
1 More Replies
ArvindDige
by New Contributor II
  • 450 Views
  • 2 replies
  • 0 kudos

Is DBFS going to be deprecated?

Is DBFS going to be deprecated? As I am using /dbfs/FileStore/tables/ location where a jar file is stored, and I am copying this jar file to /databricks/jars locations.My concerns is as DBFS root and mounts are deprecated, is that mean in coming days...

  • 450 Views
  • 2 replies
  • 0 kudos
Latest Reply
ArvindDige
New Contributor II
  • 0 kudos

Hi Raphael,I am trying below init script to achieve this task, PFAAnd getting error as below,Cluster scoped init script abfss://container@storage.dfs.core.windows.net/init_script.sh failed: Failure to initialize configuration for storage account stor...

  • 0 kudos
1 More Replies
Prashanthkumar
by New Contributor III
  • 3992 Views
  • 7 replies
  • 0 kudos

Is it possible to view Databricks cluster metrics using REST API

I am looking for some help on getting databricks cluster metrics such as memory utilization, CPU utilization, memory swap utilization, free file system using REST API.I am trying it in postman using databricks token and with my Service Principal bear...

Prashanthkumar_0-1705104529507.png
  • 3992 Views
  • 7 replies
  • 0 kudos
Latest Reply
javierbg
New Contributor III
  • 0 kudos

At my company we are also interested in this feature, is there an ETA?

  • 0 kudos
6 More Replies
rt-slowth
by Contributor
  • 661 Views
  • 2 replies
  • 0 kudos

How to update python's runtime on AWS lambda function

I heard that version 3.8 of Python on AWS Lambda will be EOL within the year. I would like to update this runtime, but where can I find the CloundFormation stack template.

  • 661 Views
  • 2 replies
  • 0 kudos
Latest Reply
sandipkumar
New Contributor II
  • 0 kudos

Thanks. I went to AWS Cloudformation stack and edited the template from python 3.8 to 3.12 and updated. I did this for both the workspace stack and the s3 ingestion stack. Will it break anything? Do I need to make any changes in the python code in th...

  • 0 kudos
1 More Replies
yeungcase
by New Contributor II
  • 158 Views
  • 2 replies
  • 0 kudos

Is it possible to configure a DLT pipeline continue to run when a single table ingestion failed?

Is it possible to configure a DLT pipeline continue to run when a single table ingestion failed?

  • 158 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @yeungcase,  When working with Delta Live Tables (DLT), you can configure the RETRY_ON_FAILURE property to allow a pipeline to continue running even if a single table ingestion fails1. This can be useful for maintaining data consistency and ensuri...

  • 0 kudos
1 More Replies
kiranpeesa
by New Contributor
  • 216 Views
  • 1 replies
  • 0 kudos

Error in notebook while execution

Error in callback <bound method UserNamespaceCommandHook.post_run_cell of <dbruntime.DatasetInfo.UserNamespaceCommandHook object at 0x7f5790c07070>> (for post_run_cell)

  • 216 Views
  • 1 replies
  • 0 kudos
Latest Reply
Witold
New Contributor III
  • 0 kudos

https://community.databricks.com/t5/data-engineering/error-in-notebook-execution/m-p/76226#M35165

  • 0 kudos
Henrik_
by New Contributor
  • 4839 Views
  • 3 replies
  • 0 kudos

Callback bound method error

 When executing a withColumn (running on DBR 14.3 LST) I get this error:Error in callback <bound method UserNamespaceCommandHook.post_run_cell of <dbruntime.DatasetInfo.UserNamespaceCommandHook object at 0x7feda2b2efb0>> (for post_run_cell):How shoul...

  • 4839 Views
  • 3 replies
  • 0 kudos
Latest Reply
TjommeV-Vlaio
New Contributor II
  • 0 kudos

We have the same issue using a shared cluster running DBR 14.3:Code executed: dfNew = dfTmp.withColumn(HashKeyColumnName, F.sha2(F.concat_ws("||", *ColumnList), 256))Error received: Error in callback <bound method UserNamespaceCommandHook.post_run_ce...

  • 0 kudos
2 More Replies
Zavi
by New Contributor
  • 327 Views
  • 1 replies
  • 0 kudos

When are DLT going to support multiple targets

Due to the limitations with all output data needing to be stored in one target we have stopped using DLT until more flexibility is added. If anyone has a workaround we are open to suggestions. 

  • 327 Views
  • 1 replies
  • 0 kudos
Latest Reply
Rafael-Ribeiro
New Contributor II
  • 0 kudos

Hi Zavi,One potential workaround is to establish multiple DLT pipelines, with each pipeline specifically configured to point to a unique target. This approach effectively allows for a diverse range of output data to be stored across various targets.T...

  • 0 kudos
nikhilprajapati
by New Contributor
  • 714 Views
  • 2 replies
  • 1 kudos

Data in dataframe is also getting deleted when we are trying to delete records from underlying table

  Hi , We are trying to load data from a delta table to a dataframe(a copy of original table) . Initially delta table has count 911 . The dataframe in which the data is loaded also has the same count .Now,  we are deleting some records from the delta...

nikhilprajapati_1-1701930598953.png nikhilprajapati_2-1701930598960.png nikhilprajapati_3-1701930598967.png nikhilprajapati_4-1701930598974.png
  • 714 Views
  • 2 replies
  • 1 kudos
Latest Reply
Hkesharwani
Contributor II
  • 1 kudos

Hi, There is a way to retain the copy of data frame, even if the data in underling table is manipulated but that's a memory expensive operation, be careful while using it.df1 = spark.createDataFrame(df.rdd.map(lambda x: x), schema=df.schema)Here we a...

  • 1 kudos
1 More Replies
Vanshika
by New Contributor
  • 136 Views
  • 1 replies
  • 1 kudos

Databricks and Cloud Services Pricing

Hi,If I connect databricks (trial version) with AWS/Azure/Google Cloud and then work on dashboards and Genie - will there be any minimal charges, or its completely free to use the cloud services?

  • 136 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Vanshika, When you sign up for the Databricks trial, you can test-drive the full Databricks platform free for 14 days on your choice of AWS, Microsoft Azure, or Google Cloud1. During this trial period, you’ll have access to all the features, i...

  • 1 kudos
mearupmukherjee
by New Contributor II
  • 325 Views
  • 2 replies
  • 1 kudos

Databricks Certification Test centre Examination Request

Hi Databricks Team,I would like to kindly request the addition of an offline option (Test centre Examination) for the Databricks certification.With the current technical issues that have been occurring, I am now hesitant and anxious about taking the ...

  • 325 Views
  • 2 replies
  • 1 kudos
Latest Reply
Cert-Team
Honored Contributor III
  • 1 kudos

Using online proctor keeps the cost for exams lower, but we can look into this option.

  • 1 kudos
1 More Replies
karola61
by New Contributor II
  • 502 Views
  • 2 replies
  • 1 kudos

Resolved! org.apache.spark.SparkException: Job aborted due to stage failure:

org.apache.spark.SparkException: Job aborted due to stage failure:

  • 502 Views
  • 2 replies
  • 1 kudos
Latest Reply
rajeshg
New Contributor II
  • 1 kudos

Along with Job aborted due to stage failure: if you see slave lost... then it is due to less memory allocated for executors, more cores per executor more memory required or the other possibility is you have used max cpu available in cluster and the d...

  • 1 kudos
1 More Replies
fperry
by New Contributor
  • 143 Views
  • 1 replies
  • 0 kudos

Concurrent State Update from Worker Nodes Possible?

For a data processing pipeline I use structured streaming and arbitrary stateful processing. I was wondering if the partitioning over several worker nodes and thus updating the state from different worker nodes has to be considered (e.g. using a lock...

  • 143 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @fperry, When using applyInPandasWithState in PySpark, updates to each group’s state are automatically saved across invocations1. The function you provide should take parameters (key, Iterator[pandas.DataFrame], state) and return another Iterator[...

  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors