cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

leelee3000
by Esteemed Contributor III
  • 2517 Views
  • 4 replies
  • 2 kudos

Resolved! Setting up Unity Catalog in Azure

Trying to create a metastore that will be connected to an external storage (ADLS) but we don't have the option to create a new metastore in 'Catalog' tab in the UI. Based on some research, we see that we'll have to go into "Manage Account" and then c...

  • 2517 Views
  • 4 replies
  • 2 kudos
Latest Reply
bsadler
New Contributor II
  • 2 kudos

I have been wrestling with this question for days now.  I seem to be the only one with this question so I am sure I am doing something wrong.  I am trying to create a UC metastore but there is not an option in "Catalog" to create a metastore.  This s...

  • 2 kudos
3 More Replies
eheinlein
by New Contributor
  • 325 Views
  • 1 replies
  • 0 kudos

How to confirm a workspace ID via an api token?

Hello! We are integrating with Databricks and we get the API key, workspace ID, and host from our users in order to connect to Databricks. We need the to validate the workspace ID because we do need it outside of the context of the API key (with webh...

  • 325 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @eheinlein, You can obtain the workspace ID from within a Databricks Notebook by running the following command in a Python or Scala cell: spark.conf.get("spark.databricks.clusterUsageTags.clusterOwnerOrgId") This command will return the worksp...

  • 0 kudos
samsonite
by New Contributor
  • 412 Views
  • 1 replies
  • 0 kudos

Failed deploying bundle via gitlab - Request failed for POST

I'm encountering an issue in my .gitlab-ci.yml file when attempting to execute databricks bundle deploy -t prod. The error message I receive is: Error: Request failed for POST <path>/state/deploy.lockInterestingly, when I run the same command locally...

  • 412 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @samsonite, The error message you’re encountering seems related to a lock request.  Ensure that the credentials and permissions used for deploying to the production environment are correct. Differences in credentials between local and CI/CD enviro...

  • 0 kudos
Ramakrishnan83
by New Contributor III
  • 11384 Views
  • 7 replies
  • 0 kudos

Renaming the database Name in Databricks

Team,Initially our team created the databases with the environment name appended. Ex: cust_dev, cust_qa, cust_prod.I am looking to standardize the database name as consistent name across environments. I want to rename to "cust". All of my tables are ...

  • 11384 Views
  • 7 replies
  • 0 kudos
Latest Reply
Avvar2022
Contributor
  • 0 kudos

You can also use “CASCADE” to drop schema and tables as well. It is recursive. 

  • 0 kudos
6 More Replies
df_Jreco
by New Contributor II
  • 377 Views
  • 1 replies
  • 0 kudos

Custom python package iin Notebook task using bundle

Hi mates!I'n my company, we are moving our pipelines to Databricks bundles, our pipelines use a notebook that receives some parameters.This notebook uses a custom python package to apply the business logic based on the parameters that receive.The thi...

  • 377 Views
  • 1 replies
  • 0 kudos
Latest Reply
df_Jreco
New Contributor II
  • 0 kudos

 Solved understanding the databricks.yml configuration!

  • 0 kudos
pranay
by New Contributor
  • 392 Views
  • 1 replies
  • 0 kudos

Data Ingestion into DLT from Azure Event hub batch processing

I am building my first DLT pipeline and I want to ingest data from Azure event hub for batch processing.But, I can just see documentation for streaming by using kafka.Can we do batch processing with DLT & Azure Event hub?

  • 392 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @pranay,  Since Databricks Notebooks allow you to run Python code, you can leverage Python libraries to manipulate Excel files.Instead of using pywin32, consider using libraries like pandas or openpyxl to read, modify, and save Excel files.You can...

  • 0 kudos
Awoke101
by New Contributor III
  • 357 Views
  • 1 replies
  • 0 kudos

UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION in shared access mode?

I'm using a shared access cluster and am getting this error while trying to upload to Qdrant.  #embeddings_df = embeddings_df.limit(5) options = { "qdrant_url": QDRANT_GRPC_URL, "api_key": QDRANT_API_KEY, "collection_name": QDRANT_COLLEC...

Community Platform Discussions
qdrant
shared_acess
UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION
  • 357 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Awoke101,  Shared access clusters in Databricks have certain restrictions due to Unity Catalog limitations.I recommend trying the single-user Unity Catalog cluster or configuring data persistence outside the container. Let me know if you need fur...

  • 0 kudos
kellenpink
by New Contributor
  • 284 Views
  • 1 replies
  • 0 kudos

How does Datbricks differ from snowflake with respect to AI tooling

How does Datbricks differ from snowflake with respect to AI tooling

  • 284 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @kellenpink, Databricks and Snowflake are both powerful platforms, but they excel in different areas:   Databricks shines in big data processing, machine learning, and AI workloads. It provides an integrated environment for data enginee...

  • 0 kudos
kellenpink
by New Contributor
  • 281 Views
  • 1 replies
  • 0 kudos

Hi, how can one go about assessing the value created due to the implementation of DBRX?

Hi, how can one go about assessing the value created due to the implementation of DBRX?

  • 281 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @kellenpink, Assessing the value created by implementing Databricks (DBRX) involves several considerations. Here are some steps you can take: Business Objectives and KPIs: Start by defining clear business objectives for using Databricks. What ...

  • 0 kudos
esi
by New Contributor
  • 4200 Views
  • 2 replies
  • 0 kudos

numpy.ndarray size changed, may indicate binary incompatibility

Hi All,I have installed the following libraries on my cluster (11.3 LTS that includes Apache Spark 3.3.0, Scala 2.12):numpy==1.21.4flair==0.12 ‎on executing `from flair.models import TextClassifier`, I get the following error:"numpy.ndarray size chan...

  • 4200 Views
  • 2 replies
  • 0 kudos
Latest Reply
sean_owen
Esteemed Contributor III
  • 0 kudos

You have changed the numpy version, and presumably that is not compatible with other libraries in the runtime. If flair requires later numpy, then use a later DBR runtime for best results, which already has later numpy versions

  • 0 kudos
1 More Replies
niruban
by New Contributor II
  • 472 Views
  • 1 replies
  • 0 kudos

Databricks Asset Bundle Behaviour for new workflows and existing workflows

Dear Community Members -I am trying to deploy a workflow using DAB. After deploying if I am updating the same workflow with different bundle name it is creating a new workflow instead of updating the existing workflow. Also when I am trying to use sa...

  • 472 Views
  • 1 replies
  • 0 kudos
Latest Reply
niruban
New Contributor II
  • 0 kudos

@nicole_lu_PM : Do you have any suggestions or feedback for the above question ? It will be really helpful if we can get some insights. 

  • 0 kudos
pdemeulenaer
by New Contributor
  • 544 Views
  • 1 replies
  • 0 kudos

Databricks asset bundles dependencies

Is anyone aware of a way to include a requirements.txt within the job definition of a databricks asset bundle? Documentation mentions how to have dependencies in workspace files, or Unity Catalog volumes, but I wanted to ask if it is possible to decl...

Community Platform Discussions
databricksassetbundles
Dependency
  • 544 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @pdemeulenaer, When working with Databricks Asset Bundles, you can specify library dependencies for local development in the requirements*.txt file at the root of the bundle project. However, job task library dependencies are declared in your bund...

  • 0 kudos
PragyaS
by New Contributor
  • 646 Views
  • 1 replies
  • 0 kudos

Creating Zip file on Azure storage explorer

I need to create zip file which will contain csv files which I have created from the dataframe. But I am unable to create valid zip file which will be containing all csv files. Is it possible to create zip file from code in databricks on azure storag...

  • 646 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @PragyaS,  References: Creating Zip file on Azure storage explorer - Databricks CommunityHow to zip files (on Azure Blob Storage) with shutil in DatabricksWorking with Unity Catalog in Azure DatabricksAnnouncing the General Availability of Unity C...

  • 0 kudos
nileshtiwaari
by New Contributor
  • 333 Views
  • 1 replies
  • 0 kudos

Spark structured streaming

hi,could someone please help me with this code :-input parameter df is a spark structured streaming dataframe def apply_duplicacy_check(df, duplicate_check_columns):    if len(duplicate_check_columns) == 0:         return None, df    valid_df = df.dr...

  • 333 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @nileshtiwaari , The error message you’re encountering indicates that the .exceptAll() operation is not supported on streaming DataFrames. In structured streaming, certain operations have limitations due to the nature of streaming data. To addr...

  • 0 kudos
bamvallar
by New Contributor
  • 317 Views
  • 1 replies
  • 0 kudos
  • 317 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @bamvallar, Welcome to the Databricks Community! We're thrilled to have you here and thank you for attending the DAIS and visiting our community booth. Your participation means a lot to us, and we're excited to see the contributions you'll bring ...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors