- 3021 Views
- 2 replies
- 1 kudos
Data in dataframe is also getting deleted when we are trying to delete records from underlying table
Hi , We are trying to load data from a delta table to a dataframe(a copy of original table) . Initially delta table has count 911 . The dataframe in which the data is loaded also has the same count .Now, we are deleting some records from the delta...
- 3021 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, There is a way to retain the copy of data frame, even if the data in underling table is manipulated but that's a memory expensive operation, be careful while using it.df1 = spark.createDataFrame(df.rdd.map(lambda x: x), schema=df.schema)Here we a...
- 1 kudos
- 25413 Views
- 6 replies
- 0 kudos
Renaming the database Name in Databricks
Team,Initially our team created the databases with the environment name appended. Ex: cust_dev, cust_qa, cust_prod.I am looking to standardize the database name as consistent name across environments. I want to rename to "cust". All of my tables are ...
- 25413 Views
- 6 replies
- 0 kudos
- 0 kudos
You can also use “CASCADE” to drop schema and tables as well. It is recursive.
- 0 kudos
- 1481 Views
- 0 replies
- 0 kudos
How to confirm a workspace ID via an api token?
Hello! We are integrating with Databricks and we get the API key, workspace ID, and host from our users in order to connect to Databricks. We need the to validate the workspace ID because we do need it outside of the context of the API key (with webh...
- 1481 Views
- 0 replies
- 0 kudos
- 1651 Views
- 1 replies
- 0 kudos
Custom python package iin Notebook task using bundle
Hi mates!I'n my company, we are moving our pipelines to Databricks bundles, our pipelines use a notebook that receives some parameters.This notebook uses a custom python package to apply the business logic based on the parameters that receive.The thi...
- 1651 Views
- 1 replies
- 0 kudos
- 0 kudos
Solved understanding the databricks.yml configuration!
- 0 kudos
- 1572 Views
- 0 replies
- 0 kudos
Spot label in pool even though the configuration selected is all on-demand
Why is there a spot label in pool even though the configuration selected is all on-demand? can someone explain?
- 1572 Views
- 0 replies
- 0 kudos
- 815 Views
- 0 replies
- 0 kudos
Is there any specific error you are receiving when running the init script? Does the run complete st
Is there any specific error you are receiving when running the init script? Does the run complete start up or fail due to the init script?
- 815 Views
- 0 replies
- 0 kudos
- 957 Views
- 0 replies
- 0 kudos
Eorror in perform a merge inside a streaming foreachbatch using the command: microBatchDF._jdf.s
I'm trying to perform a merge inside a streaming foreachbatch using the command: microBatchDF._jdf.sparkSession().sql(self.merge_query)Streaming
- 957 Views
- 0 replies
- 0 kudos
- 1250 Views
- 0 replies
- 0 kudos
ipywidgets not loading in DataBricks Community Edition
Hi All,I am trying to run commands with ipywidgets but it just says :Loading widget...Same error occurs even when I re run the cell.DataBricks version used : 14.2
- 1250 Views
- 0 replies
- 0 kudos
- 16950 Views
- 2 replies
- 1 kudos
numpy.ndarray size changed, may indicate binary incompatibility
Hi All,I have installed the following libraries on my cluster (11.3 LTS that includes Apache Spark 3.3.0, Scala 2.12):numpy==1.21.4flair==0.12 on executing `from flair.models import TextClassifier`, I get the following error:"numpy.ndarray size chan...
- 16950 Views
- 2 replies
- 1 kudos
- 1 kudos
You have changed the numpy version, and presumably that is not compatible with other libraries in the runtime. If flair requires later numpy, then use a later DBR runtime for best results, which already has later numpy versions
- 1 kudos
- 2255 Views
- 1 replies
- 0 kudos
Databricks Asset Bundle Behaviour for new workflows and existing workflows
Dear Community Members -I am trying to deploy a workflow using DAB. After deploying if I am updating the same workflow with different bundle name it is creating a new workflow instead of updating the existing workflow. Also when I am trying to use sa...
- 2255 Views
- 1 replies
- 0 kudos
- 0 kudos
@nicole_lu_PM : Do you have any suggestions or feedback for the above question ? It will be really helpful if we can get some insights.
- 0 kudos
- 1898 Views
- 0 replies
- 0 kudos
Restore committed changes to Azure Databricks Git after abandoned pull request
I want to restore the committed changes (before and after view) in my branch. As this pull request was abandoned in Azure DevOps, then the branch was not merged. Therefore, the modified notebooks still exist but not the commits.How can I retrieve aga...
- 1898 Views
- 0 replies
- 0 kudos
- 866 Views
- 0 replies
- 0 kudos
Hi, how can one go about assessing the value created due to the implementation of DBRX?
Hi, how can one go about assessing the value created due to the implementation of DBRX?
- 866 Views
- 0 replies
- 0 kudos
- 992 Views
- 0 replies
- 0 kudos
How does Datbricks differ from snowflake with respect to AI tooling
How does Datbricks differ from snowflake with respect to AI tooling
- 992 Views
- 0 replies
- 0 kudos
- 2112 Views
- 1 replies
- 0 kudos
org.apache.spark.SparkException: Job aborted due to stage failure:
org.apache.spark.SparkException: Job aborted due to stage failure:
- 2112 Views
- 1 replies
- 0 kudos
- 0 kudos
Along with Job aborted due to stage failure: if you see slave lost... then it is due to less memory allocated for executors, more cores per executor more memory required or the other possibility is you have used max cpu available in cluster and the d...
- 0 kudos
- 1434 Views
- 0 replies
- 0 kudos
UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION in shared access mode?
I'm using a shared access cluster and am getting this error while trying to upload to Qdrant. #embeddings_df = embeddings_df.limit(5) options = { "qdrant_url": QDRANT_GRPC_URL, "api_key": QDRANT_API_KEY, "collection_name": QDRANT_COLLEC...
- 1434 Views
- 0 replies
- 0 kudos
