cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

T_1
by New Contributor III
  • 24826 Views
  • 13 replies
  • 3 kudos

Resolved! displayHTML can't seem to be used from Python code, only hand typed into a cell???

Trying to use displayHTML from w/in a Python module gets a Python exception:NameError: name 'displayHTML' is not definedand I've found no way around this. It seems to be something at the UI layer or something, not a Python function that can be refere...

  • 24826 Views
  • 13 replies
  • 3 kudos
Latest Reply
T_1
New Contributor III
  • 3 kudos

Holy Guacamole Batman! It works finally!!!! Wow, thanks @ptweir That's awesome! I can go back and update my doc (and code, to just use databricks the same, now, and Jupyter!) and it'll work by default. It's great they fixed it, shame they never told ...

  • 3 kudos
12 More Replies
Data_Engineer_3
by New Contributor III
  • 16385 Views
  • 17 replies
  • 7 kudos

Resolved! FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The data and file exists in location mentioned above

I am new to learning Spark and working on some practice; I have uploaded a zip file in DBFS /FileStore/tables directory and trying to run a python code to unzip the file; The python code is as: from zipfile import *with ZipFile("/FileStore/tables/fli...

  • 16385 Views
  • 17 replies
  • 7 kudos
Latest Reply
883022
New Contributor II
  • 7 kudos

What if changing the runtime is not an option? I'm experiencing a similar issue using the following:%pip install -r /dbfs/path/to/file.txtThis worked for a while, but now I'm getting the Errno 2 mentioned above. I am still able to print the same file...

  • 7 kudos
16 More Replies
Oliver_Angelil
by Valued Contributor II
  • 6552 Views
  • 4 replies
  • 0 kudos

Resolved! Python code linter in Databricks notebook

Is it possible to get syntax linting in a DB notebook? Say with flake8, like I do in VS code?

  • 6552 Views
  • 4 replies
  • 0 kudos
Latest Reply
artsheiko
Honored Contributor
  • 0 kudos

No linting in a DB notebook available for now. The Notebook is currently in the process of adopting Monaco as the underlying code editor which will offer an improved code authoring experience for notebook cells.Some of the Monaco editor features enab...

  • 0 kudos
3 More Replies
KVNARK
by Honored Contributor II
  • 1068 Views
  • 1 replies
  • 4 kudos

Resolved! a usecase to query millions of values.

have a small use case where we need to query the sql database with 1 million values(dynamically returned from python code) in the condition from python function. eg: select * from id in (1,2,23,33........1M). I feel this is very bad approach. Is ther...

  • 1068 Views
  • 1 replies
  • 4 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 4 kudos

You can also create a temporary view with the output from python code (one id = one row) and then inner join the view to the table. IMO will improve readability of your code.

  • 4 kudos
aben1
by New Contributor
  • 1126 Views
  • 1 replies
  • 0 kudos

I have created a piece of python code, which lead to some python error.The job have failed with Internal Error, see below.The message after clicking o...

I have created a piece of python code, which lead to some python error.The job have failed with Internal Error, see below.The message after clicking on it states somewhat miseleading info:Meanwhile the real issue is fortunatelly described in Logs I d...

image image image
  • 1126 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @a Ben​, Do you still need help, or is your issue resolved?

  • 0 kudos
kirankv
by New Contributor
  • 694 Views
  • 0 replies
  • 0 kudos

How to get notebookid programmatically using R

Hi, I would like to log the notebook id programmatically in R, Is there any command that exists in R so that I can leverage to grab the notebook id, I tried with python using the below command and grab it without any issues, and looking for similar f...

  • 694 Views
  • 0 replies
  • 0 kudos
gbrueckl
by Contributor II
  • 4734 Views
  • 6 replies
  • 4 kudos

Resolved! CREATE FUNCTION from Python file

Is it somehow possible to create an SQL external function using Python code?the examples only show how to use JARshttps://docs.databricks.com/spark/latest/spark-sql/language-manual/sql-ref-syntax-ddl-create-function.htmlsomething like:CREATE TEMPORAR...

  • 4734 Views
  • 6 replies
  • 4 kudos
Latest Reply
pts
New Contributor II
  • 4 kudos

As a user of your code, I'd find it a less pleasant API because I'd have to some_module.some_func.some_func() rather than just some_module.some_func()No reason to have "some_func" exist twice in the hierarchy. It's kind of redundant. If some_func is ...

  • 4 kudos
5 More Replies
Development
by New Contributor III
  • 610 Views
  • 0 replies
  • 0 kudos

Hi All, I hope you're doing well I am facing issue while installing an python library on ADB Cluster. lib - PyCaret ( latest version) its not gett...

Hi All,I hope you're doing wellI am facing issue while installing an python library on ADB Cluster.lib - PyCaret ( latest version)its not getting install and showing me 'Failed' Status.It would be great if you can help here !!Thanks

  • 610 Views
  • 0 replies
  • 0 kudos
Hubert-Dudek
by Esteemed Contributor III
  • 14902 Views
  • 11 replies
  • 12 kudos

Resolved! dbutils or other magic way to get notebook name or cell title inside notebook cell

Not sure it exists but maybe there is some trick to get directly from python code:NotebookNameCellTitlejust working on some logger script shared between notebooks and it could make my life a bit easier

  • 14902 Views
  • 11 replies
  • 12 kudos
Latest Reply
jose_gonzalez
Moderator
  • 12 kudos

Hi @Hubert Dudek​ ,The following command will print the path where your notebook is located. If you split the returned string and only select the last string, then you will be able to get the notebook namenotebook_name = dbutils.notebook.entry_point....

  • 12 kudos
10 More Replies
Nickels
by New Contributor II
  • 1836 Views
  • 4 replies
  • 1 kudos

Resolved! Reply on inline runtime commands

I feel like the answer to this question should be simple, but none the less I'm struggling.I run a python code that prompts me with the following warning:On my local machine, I can accept this through my terminal and my machine do not run out of memo...

image
  • 1836 Views
  • 4 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Moderator
  • 1 kudos

Hi @Nickels Köhling​ ,In Databricks, you will only be able to see the output in the driver logs. If you go to your driver logs, you will be able to see 3 windows that are displaying the output of "stdout", "stderr" and "log4j".If in your code you do ...

  • 1 kudos
3 More Replies
User16826994223
by Honored Contributor III
  • 1658 Views
  • 1 replies
  • 0 kudos

Resolved! How to find best model using python in mlflow

I have a use case in mlflow with python code to find a model version that has the best metric (for instance, “accuracy”) among so many versions , I don't want to use web ui but to use python code to achieve this. Any Idea?

  • 1658 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

import mlflow client = mlflow.tracking.MlflowClient() runs = client.search_runs("my_experiment_id", "", order_by=["metrics.rmse DESC"], max_results=1) best_run = runs[0]https://mlflow.org/docs/latest/python_api/mlflow.tracking.html#mlflow.tracking.M...

  • 0 kudos
Labels